• Viri4thus@feddit.org
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 month ago

      I also have a 3060, can you detail which framework (sglang, ollama, etc) you are using and how you got that speed? i’m having trouble reaching that level of performance. Thx

    • levzzz@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 month ago

      You need a pretty large context window to fit all the reasoning, ollama forces 2048 by default and more uses more memory