Update 7/10/2024 and 8/10/2024

Analysis and comparisson of libraries:

1. Windows Event Log API* (EvtQuery, EvtNext, etc.) :

   • Older and more traditional C-style approach
   • Part of the Windows SDK
   • Provides direct access to event logs
   • Can be used with C++, but may require more wrapper code for modern C++ integration
   • Pros: Well-documented, stable, and widely supported
   • Cons: More verbose, may require more boilerplate code in C++

2. Event Tracing for Windows (ETW):

   • More modern approach to event tracing and logging
   • Provides real-time and historical event collection
   • Offers high-performance event tracing
   • Can be used with C++ and integrates well with modern C++ features
   • Pros: High performance, flexible, supports real-time monitoring
   • Cons: Steeper learning curve, may be overkill for simple event log queries

3. krabsetw:

   • Open-source C++ library for ETW
   • Provides a modern C++ wrapper around ETW functionality
   • Designed to work well with C++11 and later
   • Pros: Easier to use than raw ETW, good C++ integration
   • Cons: Third-party library, may have less documentation compared to official Microsoft solutions

4. Integrating PowerShell:

   • Allows using PowerShell cmdlets from C++ code
   • Can leverage PowerShell's Get-WinEvent cmdlet for querying event logs
   • Requires setting up PowerShell hosting in C++
   • Pros: Flexible, can use PowerShell's rich event querying capabilities
   • Cons: Additional overhead of PowerShell runtime, may be slower for large-scale querying

First Approach

Some basic design and tentative approach on how can this be achieved, this should be double checked based on the different libraries limitations and needs:

Class Diagram:

classDiagram
    class EventChannelQuerySystem {
        - Session session
        - Provider provider
        - Consumer consumer
        -BookmarkManager bookmarkMgr
        -QueryBuilder queryBuilder
        +initialize()
        +startQuery(channelName, queryString)
        +stopQuery()
        +getResults()
        +setReconnectTime(time)
        +rotateLog()
    }
    class Session {
        -sessionName
        -sessionProperties
        +open()
        +close()
        +enableProvider(provider)
    }
    class Provider {
    }
    class Consumer {
        -eventCallback
        +processEvent(event)
        +setEventCallback(callback)
    }
    class BookmarkManager {
        -bookmarks
        +createBookmark(event)
        +getBookmark(id)
        +saveBookmarks()
        +loadBookmarks()
    }
    class QueryBuilder {
        +buildQuery(channelName, conditions)
        +addWildcardMatch(field, pattern)
    }

    EventChannelQuerySystem -- Session
    EventChannelQuerySystem -- Provider
    EventChannelQuerySystem -- Consumer
    EventChannelQuerySystem -- BookmarkManager
    EventChannelQuerySystem -- QueryBuilder

1. EventChannelQuerySystem: The main class that orchestrates the entire querying process.
2. Session: Manages the ETW session, including opening, closing, and enabling providers.
3. Provider: Represents an ETW provider, which can be enabled or disabled.
4. Consumer: Processes the events received from the ETW session.
5. BookmarkManager: Handles the creation, saving, and loading of bookmarks.
6. QueryBuilder: Constructs queries, including support for wildcard matches.

Flow chart:

1. The system starts by initializing the ETWQuerySystem.
2. It opens an ETW session and enables the required provider.
3. The ETW consumer is set up to handle incoming events.
4. Bookmarks are loaded from persistent storage.
5. A query is built using the QueryBuilder, incorporating channel name and specific query conditions.
6. The query is started, and the system enters an event processing loop.
7. For each received event:
   • The event is processed.
   • Bookmarks are updated.
   • Log rotation is performed if needed.
8. The system checks if reconnection is needed and performs it if necessary.
9. When the query is stopped, bookmarks are saved, and the session is closed.

graph TD
    A[init] --> B[Initialize - ETWQuerySystem]
    B --> C[Open - ETWSession]
    C --> D[Enable - ETWProvider]
    D --> E[Set up - ETWConsumer]
    E --> F[Load Bookmarks -> store]
    E --> W[(Store)]
    F --> G[Build N Queries]
    G --> H[run]
    H --> I{While Event Received?}
    I -->|Yes| J[Process Event]
    J --> K[Update Bookmark]
    K --> I
    I -->|No| N{Query Stopped or shutdown signal ?}
    N -->|Yes| O[Save Bookmarks]
    O --> P[Close Session]
    P --> Q[End]
    N -->|No| R{Reconnect Needed?}
    R -->|Yes| S[Reconnect]
    S --> I
    R -->|No| I

Update 9/10/2024

• Working on a base example for getting events in real-time on windows using ETW.
• Poor and hard to follow documentation.
• hard to reuse examples.
• Getting some base events. but looking for an alternative.

Update 10/10/2024

• Working on krabsetw aproach:

  • Better documented and easy to find example on git.
  • Easy to use and to integrate with vcpkg.
  • Still working on getting events from any provider.

  Code using krabsetw can be as simple as:

    krabs::user_trace trace;
    krabs::provider<> provider(krabs::guid(L"{A0C1853B-5C40-4B15-8766-3CF1C58F985A}"));
    provider.any(0xf0010000000003ff);
    provider.add_on_event_callback([](const EVENT_RECORD &record, const krabs::trace_context &trace_context) {
        krabs::schema schema(record, trace_context.schema_locator);
        std::wcout << L"Event " << schema.event_id();
        std::wcout << L"(" << schema.event_name() << L") received." << std::endl;
        if (schema.event_id() == 7937) {
            krabs::parser parser(schema);
            std::wstring context = parser.parse<std::wstring>(L"ContextInfo");
            std::wcout << L"\tContext: " << context << std::endl;
        }
    });
  
    trace.enable(provider);
    trace.start();

Update 14/10/2024

Seccond Approach

Because we're using krabsetw and it needs to create a thread for each provider, this is the rework of the design:

classDiagram
    class EventChannelQuerySystem {
        - Provider provider
        - Consumer consumer
        -BookmarkManager bookmarkMgr
        -QueryBuilder queryBuilder
        +initialize()
        +setEventCallback(callback)
        +start()
        +qttyOfResults()
        +setReconnectTime(time)
        +rotateLog()
    }
    class Provider {
        -channel
        -providerName
    }
    class Consumer {
        -eventCallback
        -queryString
        +processEvent(event)
    }
    class BookmarkManager {
        -bookmarks
        +createBookmark(event)
        +getBookmark(id)
        +saveBookmarks()
        +loadBookmarks()
    }
    class QueryBuilder {
        +buildQuery(channelName, conditions)
        +addWildcardMatch(field, pattern)
    }

    EventChannelQuerySystem -- Session
    EventChannelQuerySystem -- Provider
    EventChannelQuerySystem -- Consumer
    EventChannelQuerySystem -- BookmarkManager
    EventChannelQuerySystem -- QueryBuilder

Flow chart Changes:

1. For loop for every channel neeeded
2. In each case a thread is created for that purpose.
3. Process events should fire an async task to the logcollector module.

flowchart TD
    A["init"] --> C["Initialize Channel / Provider"]
    C --> E["Initialice Trace Session"]
    E --> F["Load Bookmarks"] & W[("Store")]
    F --> G["Build / Check Queries"]
    G --> H["Start Provider Thread"]
    H --> I{"Receiveing events"} & B["Next Provider"]
    I -- Yes --> J["Process Event"]
    J --> K["Update Bookmark"]
    K --> I
    I -- No --> N{"Stopped ?"}
    N -- Yes --> O["Save Bookmarks"]
    O --> P["Close Session"]
    P --> Q["End"]
    N -- No --> R{"Reconnection?"}
    R -- Yes --> S["Reconnect"]
    S --> I
    R -- No --> I
    B --> A
    I --> n1["Untitled Node"]

Example of reading Application events -> ⚠️ User should be SYSTEM:

void EventLogApplication::start()
{
    // While Adminstrator is sufficent to view the Security EventLog,
    // SYSTEM is required for the Microsoft-Windows-Security-Auditing provider.
    char user_name[128] = { 0 };
    DWORD user_name_length = 128;
    if (!GetUserNameA(user_name, &user_name_length) || !strcmp(user_name, "SYSTEM") == 0)
    {
        std::wcout << L"Microsoft-Windows-Security-Auditing can only be traced by SYSTEM" << std::endl;
        return;
    }

    krabs::user_trace trace(L"EventLog-Application");
    krabs::provider<> provider(L"Microsoft-Windows-Security-SPP");
    provider.any((ULONGLONG)-1);
    provider.add_on_event_callback([](const EVENT_RECORD &record, const krabs::trace_context &trace_context) {
        krabs::schema schema(record, trace_context.schema_locator);
        std::wcout << L"Event " << schema.event_id();
        std::wcout << L"(" << schema.event_name() << L") received." << std::endl;

        if (schema.event_id() == 4703) {
            krabs::parser parser(schema);
            std::wstring enabled_privilege_list = parser.parse<std::wstring>(L"EnabledPrivilegeList");
            std::wstring disabled_privilege_list = parser.parse<std::wstring>(L"DisabledPrivilegeList");
            std::wcout << L"\tEnabledPrivilegeList=" << enabled_privilege_list << std::endl;
            std::wcout << L"\tDisabledPrivilegeList=" << disabled_privilege_list << std::endl;
        }
    });

    trace.enable(provider);
    trace.start();
}

Update 15/10/2024

• Integrating base case development with main issue branch.
• solving some issues with vcpkg
• Adding UT for base execution.

Update 16/10/2024

• New limitation for krabsetw I'm not able to receive old events, waiting for dev's answer here. ⚠️
• Integrating base case development with main issue branch:
  • able to run tests calling harcoded methods.
  • able to receive fixed header fields (pending search for limitations in this ground). Also non header ones.
  • Pending branch on updated base. and rest of the integration with logcollector.

Update 17/10/2024

• Integrating base case development with main issue branch:
  • Rebase and analysis of changes on the base branch.
  • modifying all necessary code to work along the base branch PR.
  • stuck on unicode issue and transforming infinite loop into a awaitable.

Update 18/10/2024

• Final rebase with master.
• Creating some unit test (in progress)
• Due to lack of time to finish before the MVP deadline and based on the limitations of the krabsetw library it was decided to rework on the solution to reach a more suitable product for the future.

Moved to on hold until the release testing stage is completed.

Update 17/12

• Restart base on Logcollector development and requirements.
• Krabsetw and ETW discarded for bookmarking issue. Starting with Windows Event Log API.
• Basic flowchart:

flowchart TD
    %% Start
    A[Start evt_reader] --> B{Open Event Log Channel}

    %% Event Log Open Failure
    B -->|Fail| X1[Error: Failed to Open Channel<br>Notify User and Exit]
    B -->|Success| C{Check for Existing Bookmark}

    %% Bookmark Handling
    C -->|Bookmark Exists| D[Query Events Using Bookmark]
    C -->|No Bookmark| E[Query Events from Start]

    %% Event Processing Loop
    D --> F{Events Available?}
    E --> F{Events Available?}

    %% Processing Events
    F -->|Yes| G[Process Event]
    G --> H[Update Bookmark]
    H -->|Success| F
    H -->|Failure| X2[Warning: Failed to Save Bookmark<br>Continue Processing]

    %% Timeout Path for No Events
    F -->|No| T[Wait for Timeout]
    T --> R{Retry Query?}
    R -->|Yes| D
    R -->|Error| X3[Error: Query Failed<br>Notify User and Exit]

    %% Cleanup
    X3 --> K
    X1 --> K[Exit evt_reader]

    %% Reconnection to Retry Query
    T -->|Timeout Expired| D

Update 19/12

• Setting up environment on windows VM
• Base Files addition
• First Configuration block:

logcollector:
  enabled: true
...
  windows:
    reconnect-time: 10s
    use-bookmark: false
    - channel: Application
      query: Event[System/EventID = 4624]
    - channel: Security
      query: Event[System/EventID = 7040]

Update 20/12

• WindowsEventChannel Reader (WECReader) base class
• Base unit test

Update 23/12

• Fixing old unit tests for windows
• Rework on design

Update 24/12

• Basic functionality methods implemented
• krabsetw events generation for testing in progress

Update 02/01

• Discarded krabsetw for testing purposes, will replace by the same library instead due to some issue while setting UNICODE
• Testing several events form different channels (Application / Security / System) Manually.
  • Filtering done ok.
  • Pending check on duplication of events.
  • Pending check on received events on mock_server (maybe issues while working with utf16.

Update 03/01

• Receiving old events. Corrections:
  • Garbage char while casting between 16 and 8 bits
  • Corrections on repeated events.
  • Leaving a single coroutine per channel
  • Pending Live events are not being processed, double check correct behavior of bookmark

Update 06/01

• Corrections on live and past events done.
• Bookmark feature checked, pending corner cases and directory fix.
• Pending Stop / cleaning procedure. Some config parsing changes and ALl Unit Tests.

Update 07/01

• Bookmark checked and corrected for several channels / queries.
• Using mock_server received some events :
  • Channel Security:

Body:
{"agent":{"groups":[],"host":{"architecture":"x86_64","hostname":"WIN-D75P8GSAED8","ip":["192.168.100.205","2803:9800:9885:a8c4:e67a:905e:a7:19ff","127.0.0.1","::1"],"os":{"name":"Microsoft Windows Server 2022 Datacenter Evaluation","type":"Unknown","version":"10.0.20348.2762"}},"id":"945f59e7-12f1-4841-8241-ae5a7d59936c","name":"","type":"Endpoint","version":"5.0.0"}}
{"module":"logcollector","type":"eventchannel"}
{"event":{"created":"2025-01-07T21:42:45.516Z","module":"logcollector","original":"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-a5ba-3e3b0328c30d}'/><EventID>4625</EventID><Version>0</Version><Level>0</Level><Task>12544</Task><Opcode>0</Opcode><Keywords>0x8010000000000000</Keywords><TimeCreated SystemTime='2025-01-07T21:42:44.5217753Z'/><EventRecordID>31138</EventRecordID><Correlation ActivityID='{dc1be5e4-6057-0002-ede5-1bdc5760db01}'/><Execution ProcessID='972' ThreadID='11248'/><Channel>Security</Channel><Computer>WIN-D75P8GSAED8</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-21-2412136705-175749101-1827833236-500</Data><Data Name='SubjectUserName'>Administrator</Data><Data Name='SubjectDomainName'>WIN-D75P8GSAED8</Data><Data Name='SubjectLogonId'>0x3ecb83</Data><Data Name='TargetUserSid'>S-1-0-0</Data><Data Name='TargetUserName'>admin</Data><Data Name='TargetDomainName'>WIN-D75P8GSAED8</Data><Data Name='Status'>0xc000006d</Data><Data Name='FailureReason'>%%2313</Data><Data Name='SubStatus'>0xc0000064</Data><Data Name='LogonType'>2</Data><Data Name='LogonProcessName'>seclogo</Data><Data Name='AuthenticationPackageName'>Negotiate</Data><Data Name='WorkstationName'>WIN-D75P8GSAED8</Data><Data Name='TransmittedServices'>-</Data><Data Name='LmPackageName'>-</Data><Data Name='KeyLength'>0</Data><Data Name='ProcessId'>0x3460</Data><Data Name='ProcessName'>C:\\Windows\\System32\\svchost.exe</Data><Data Name='IpAddress'>::1</Data><Data Name='IpPort'>0</Data></EventData></Event>","provider":"syslog"},"log":{"file":{"path":"Security"}},"tags":["mvp"]}

• Channel System - Query EventID 35:

Body:
{"agent":{"groups":[],"host":{"architecture":"x86_64","hostname":"WIN-D75P8GSAED8","ip":["192.168.100.205","2803:9800:9885:a8c4:e67a:905e:a7:19ff","127.0.0.1","::1"],"os":{"name":"Microsoft Windows Server 2022 Datacenter Evaluation","type":"Unknown","version":"10.0.20348.2762"}},"id":"945f59e7-12f1-4841-8241-ae5a7d59936c","name":"","type":"Endpoint","version":"5.0.0"}}
{"module":"logcollector","type":"eventchannel"}
{"event":{"created":"2025-01-07T21:41:43.697Z","module":"logcollector","original":"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Time-Service' Guid='{06edcfeb-0fd0-4e53-acca-a6f8bbf81bcb}'/><EventID>35</EventID><Version>0</Version><Level>4</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2025-01-07T21:41:42.6967359Z'/><EventRecordID>13599</EventRecordID><Correlation/><Execution ProcessID='12848' ThreadID='8996'/><Channel>System</Channel><Computer>WIN-D75P8GSAED8</Computer><Security UserID='S-1-5-19'/></System><EventData Name='TMP_EVENT_TIME_SOURCE_CHOSEN'><Data Name='TimeSource'>time.windows.com,0x8 (ntp.m|0x8|0.0.0.0:123-&gt;40.119.6.228:123)</Data><Data Name='TimeSourceRefId'>3825628968</Data><Data Name='CurrentStratumNumber'>4</Data></EventData></Event>","provider":"syslog"},"log":{"file":{"path":"System"}},"tags":["mvp"]}

• Channel System - Query EventID 37: Generates 2 events at the same time, this can be done till 10 events are received on the same call (for now it's no modifiable but it can be done)

Body:
{"agent":{"groups":[],"host":{"architecture":"x86_64","hostname":"WIN-D75P8GSAED8","ip":["192.168.100.205","2803:9800:9885:a8c4:e67a:905e:a7:19ff","127.0.0.1","::1"],"os":{"name":"Microsoft Windows Server 2022 Datacenter Evaluation","type":"Unknown","version":"10.0.20348.2762"}},"id":"945f59e7-12f1-4841-8241-ae5a7d59936c","name":"","type":"Endpoint","version":"5.0.0"}}
{"module":"logcollector","type":"eventchannel"}
{"event":{"created":"2025-01-07T21:41:27.712Z","module":"logcollector","original":"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Time-Service' Guid='{06edcfeb-0fd0-4e53-acca-a6f8bbf81bcb}'/><EventID>37</EventID><Version>0</Version><Level>4</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2025-01-07T21:41:26.8876581Z'/><EventRecordID>13597</EventRecordID><Correlation/><Execution ProcessID='12848' ThreadID='14956'/><Channel>System</Channel><Computer>WIN-D75P8GSAED8</Computer><Security UserID='S-1-5-19'/></System><EventData Name='TMP_EVENT_TIME_SOURCE_REACHABLE'><Data Name='TimeSource'>time.windows.com,0x8 (ntp.m|0x8|0.0.0.0:123-&gt;40.119.6.228:123)</Data></EventData></Event>","provider":"syslog"},"log":{"file":{"path":"System"}},"tags":["mvp"]}
{"module":"logcollector","type":"eventchannel"}
{"event":{"created":"2025-01-07T21:41:29.571Z","module":"logcollector","original":"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Time-Service' Guid='{06edcfeb-0fd0-4e53-acca-a6f8bbf81bcb}'/><EventID>37</EventID><Version>0</Version><Level>4</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2025-01-07T21:41:28.5597879Z'/><EventRecordID>13598</EventRecordID><Correlation/><Execution ProcessID='12848' ThreadID='14956'/><Channel>System</Channel><Computer>WIN-D75P8GSAED8</Computer><Security UserID='S-1-5-19'/></System><EventData Name='TMP_EVENT_TIME_SOURCE_REACHABLE'><Data Name='TimeSource'>time.windows.com,0x8 (ntp.m|0x8|0.0.0.0:123-&gt;40.119.6.228:123)</Data></EventData></Event>","provider":"syslog"},"log":{"file":{"path":"System"}},"tags":["mvp"]}