Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1936 add api log to log all api calls #2319

Open
wants to merge 27 commits into
base: dev
Choose a base branch
from

Conversation

pournasserian
Copy link
Contributor

No description provided.

Introduce new classes and extensions to facilitate logging of HTTP requests, responses, and exceptions in a web API project.

- Add `ExceptionModel` in `ExceptionModel.cs` to capture exception details.
- Add `HeaderDictionaryExtensions` in `HeaderDictionaryExtensions.cs` to convert `IHeaderDictionary` to `Dictionary<string, string>`.
- Add `HttpLog` in `HttpLog.cs` to represent a log entry for an HTTP transaction.
- Add `HttpRequestLog` in `HttpRequestLog.cs` to capture HTTP request details.
- Add `HttpResponseLog` in `HttpResponseLog.cs` to capture HTTP response details.
@pournasserian pournasserian linked an issue Nov 29, 2024 that may be closed by this pull request
@pournasserian pournasserian self-assigned this Nov 29, 2024
@pournasserian pournasserian marked this pull request as draft November 29, 2024 17:04
@pournasserian pournasserian added the enhancement New feature or request label Nov 29, 2024
@pournasserian pournasserian added this to the MVP milestone Nov 29, 2024
Refactored `ExceptionModel`, `HttpLog`, `HttpRequestLog`, `HttpResponseLog`, and `IApiExecutionContext` to new namespace `FluentCMS.Entities.Logging`. Removed `HeaderDictionaryExtensions.cs`. Added `HttpLoggingMiddleware` for logging HTTP requests and responses. Introduced `IHttpLogRepository` and `HttpLogRepository` for logging to LiteDB.
Removed ExceptionModel and introduced HttpException. Refactored HttpLoggingMiddleware for better exception handling and logging. Updated HttpLog to use HttpException. Added HttpLogService and IHttpLogService for logging HTTP logs. Updated DI to include IHttpLogRepository service.
Added ApiTokenKey to IApiExecutionContext and ApiExecutionContext.
Refactored HttpLog class to include new properties and removed
Context and Exception properties. Introduced HttpLoggingMiddleware
to log HTTP requests and responses. Enhanced HttpLoggingMiddleware
with additional logging details. Simplified HttpLogRepository and
added it to MongoDBServiceExtensions for dependency injection.
- Modify constructor to initialize `_liteDatabase` and `_liteDbContext` fields using `liteDbContext` parameter.
- Introduce static boolean `_isInitialized` to track database initialization.
- Add check in constructor to set `_isInitialized` if database has at least one collection.
- Update `Create` method to check `_isInitialized` before log creation.
- Rename `LiteDatabase` and `LiteDbContext` fields to `_liteDatabase` and `_liteDbContext`, and change access modifiers from `protected` to `private`.
- Ensure `Create` method only inserts log if `_isInitialized` is `true`.
Added pragma directives to disable and restore the IDE0290 warning
around the constructor of the JwtAuthorizationMiddleware class. This
ensures the existing constructor format is maintained without triggering
the IDE warning.
Added a new `Url` property to `HttpLog` and renamed `DisplayUrl` to `Url` in `HttpRequestLog`. Updated `HttpLoggingMiddleware` to use the new `Url` property. Modified `HttpLogRepository` to log only if initialized. Refactored `GlobalSettingsRepository` for constructor initialization and updated methods to use the new `apiExecutionContext`.
The `HttpLoggingMiddleware` class has been modified to temporarily disable the logging of request and response bodies. The `ReadRequestBody` and `ReadResponseBody` methods have been commented out, and their invocations within the `Invoke` method have been replaced with placeholder strings. The original response stream and memory stream handling code has also been commented out. A `TODO` comment has been added to reconsider when to log the request and response bodies.
Modified the GetCollectionName method in HttpLogRepository.cs to update collection names based on status codes. Changed "HttpLog400", "HttpLog300", "HttpLog500", and "HttpLog200" to "httplog400", "httplog300", "httplog500", and "httplog200" respectively. Corrected a typo where "httphog400" should be "httplog400".
Added a static boolean field `_isInitialized` to the `HttpLogRepository` class to check if the database is initialized. This is a temporary workaround, and comments indicate uncertainty about its long-term validity.
Removed HttpException, HttpRequestLog, and HttpResponseLog classes. Refactored HttpLog class to handle request, response, and exception details directly. Updated HttpLoggingMiddleware to use IOptions for HttpLogConfig and refactored logging logic. Updated ApiServiceExtensions to bind HttpLogConfig. Added new HttpLogConfig class and updated appsettings.json for HTTP logging settings.
Moved HttpLog class to FluentCMS.Entities namespace.
Updated HttpLogService, HttpLoggingMiddleware, IHttpLogRepository,
and HttpLogRepository files to remove obsolete using directives.
Refactored HttpLoggingMiddleware to use local process variable.
Added using System.Collections directive to HttpLog.cs.
Updated README.md to include sections for "Documentation" and "Logging" with a link to the new logging documentation. Expanded HttpLog.cs with XML documentation comments for each property, capturing extensive HTTP request, response, and exception details. Added Logging.md to provide comprehensive documentation on HTTP logging, including middleware overview, configuration options, and usage examples.
A new table named `HttpLogs` has been created to store detailed logs of HTTP requests and responses. This includes metadata such as status code, duration, assembly details, process and thread information, memory usage, machine and environment details, user information, request and response details, and exception details. This addition will aid in tracking and analyzing HTTP interactions within the system.
A new table named `HttpLogs` has been created in the `Setup.sql` file to store detailed HTTP log information. This table includes columns for request and response details, user and environment information, and exception details. Additionally, the comment `-- Table: ApiTokenPolicies` was moved to a different position in the file, but the `ApiTokenPolicies` table definition remains unchanged.
Updated FluentCmsDbContext.cs to include JSON serialization and deserialization for HttpLog entity properties (ReqHeaders, ResHeaders, ExData) using JsonSerializer. Added necessary using directives for System.Collections and System.Text.Json. Ensured base class OnModelCreating method is called.
- Register `IHttpLogRepository` as a scoped service in `EFCoreServiceExtensions`.
- Add `DbSet<HttpLog>` to `FluentCmsDbContext` for managing HTTP logs.
- Update `appsettings.json` to use `sqlite` instead of `LiteDb` and adjust connection string.
- Implement `HttpLogRepository` in `FluentCMS.Repositories.EFCore` to handle HTTP log entries.
Refactored `HttpLoggingMiddleware` to use `IHttpLogService` instead of `IHttpLogRepository` for logging HTTP requests. Updated `appsettings.json` to change the database configuration from SQLite to LiteDb, including modifying the connection string to match LiteDb's format.
Refactored HttpLogService to use a background processing queue for
logging HTTP logs. Updated IHttpLogService interface to reflect
non-blocking logging. Modified HttpLoggingMiddleware to call Log
method without awaiting. Updated IHttpLogRepository interface and
repository implementations to support batch processing of log
entries, improving performance and efficiency.
@pournasserian pournasserian marked this pull request as ready for review December 1, 2024 01:04
Refactored `HttpLogService` to use `IHttpLogChannel` and removed the original implementation and interface. Commented out HTTP logging middleware in `ApiServiceExtensions.cs` and `HttpLoggingMiddleware.cs`. Updated `ServiceExtensions.cs` to comment out `BackgroundHttpLogProcessor` and `HttpLogChannel` registrations. Commented out cancellation token checks and log insertion logic in `HttpLogRepository.cs` for EFCore and LiteDb. Added `BackgroundHttpLogProcessor.cs` for batch processing of HTTP logs and `HttpLogChannel.cs` for HTTP log channel implementation.
Uncommented and restored the `HttpLogService`, `HttpLoggingMiddleware`, and `HttpLogRepository` across multiple files. Re-enabled the registration of related services and middleware in the service collection and middleware pipeline. Simplified the `ReadAllAsync` method in `HttpLogChannel.cs` by removing the try-catch block.
Updated BackgroundHttpLogProcessor and HttpLogChannel to use IOptions<HttpLogConfig> for configuration settings from appsettings.json. Modified ServiceExtensions to bind HttpLogConfig settings and removed binding from ApiServiceExtensions. Moved HttpLogConfig to FluentCMS.Services namespace and added BatchSize property. Updated appsettings.json to include BatchSize.
Replaced the `HttpLogConfig` class configuration with a new `HttpLogging` section in the `appsettings.json` file. Added a table detailing the new configuration properties, including `Enable`, `EnableRequestBody`, `EnableResponseBody`, and `BatchSize`. Updated the example JSON configuration to reflect these changes.
Refactored ApiExecutionContext to use a new constant for the API auth header key. Updated ApiResultActionFilter and ApiResultExceptionFilter to use constructor injection for IApiExecutionContext. Introduced new interfaces and classes for API results, including IApiPagingResult, ApiPagingResult, IApiResult, and ApiResult. Updated appsettings.json for database configuration and logging. Added new PersonController and Person entity. Removed AppApiClientException and added AppError and AppException classes.
Added using directive for FluentCMS.Services.Models and removed Microsoft.AspNetCore.Http directive. Simplified IApiExecutionContext registration in the DI container from a lambda expression to a direct type mapping.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add API log to log all api calls
1 participant