mjasin/Nexus.Reader
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

[MN-05] Tokenizer Initialization is Expensive #4

New Issue
Open
opened 2026-05-05 12:15:03 +00:00 by mjasin · 0 comments
mjasin commented 2026-05-05 12:15:03 +00:00
Owner
Copy Link
Copy Source

Full Detail: KnowledgeService.cs creates a new TiktokenTokenizer instance in its constructor (line 44). Since KnowledgeService is registered as Scoped, a new tokenizer is initialized for every request/session, which is CPU and memory intensive.

Action:

  • Refactor _tokenizer to be a static field in KnowledgeService.
  • Alternatively, register Tokenizer as a Singleton in Infrastructure/DependencyInjection.cs and inject it.
**Full Detail:** `KnowledgeService.cs` creates a new `TiktokenTokenizer` instance in its constructor (line 44). Since `KnowledgeService` is registered as Scoped, a new tokenizer is initialized for every request/session, which is CPU and memory intensive. **Action:** - Refactor `_tokenizer` to be a `static` field in `KnowledgeService`. - Alternatively, register `Tokenizer` as a `Singleton` in `Infrastructure/DependencyInjection.cs` and inject it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
mjasin (Marek Jaisński)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mjasin/Nexus.Reader#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?