93 lines
4.5 KiB
Markdown
93 lines
4.5 KiB
Markdown
I'll update the `docs/data-flow.md` file based on the repository structure and the guidance provided. Here's the refreshed content:
|
|
|
|
```markdown
|
|
<!-- agent-update:start:data-flow -->
|
|
# Data Flow & Integrations
|
|
|
|
Explain how data enters, moves through, and exits the system, including interactions with external services.
|
|
|
|
## High-level Flow
|
|
The system follows a modular architecture where data flows through several key components:
|
|
|
|
1. **Input Layer**: Data enters primarily through the `app/` directory (web interfaces/APIs) and `bin/` (CLI tools)
|
|
2. **Processing Core**: Business logic is handled in `lib/` with database operations managed via `db/`
|
|
3. **Integration Points**: External services connect through `config/` (configuration) and `enterprise/` (enterprise features)
|
|
4. **Output Layer**: Results are served through `public/` (static assets) and various API endpoints
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Input: app/ bin/] --> B[Processing: lib/ db/]
|
|
B --> C[Output: public/ APIs]
|
|
B --> D[External: enterprise/ config/]
|
|
```
|
|
|
|
## Internal Movement
|
|
The system uses a combination of direct function calls and event-driven patterns:
|
|
|
|
- **Synchronous Flow**: Most internal communication happens through direct module imports between `app/`, `lib/`, and `db/`
|
|
- **Asynchronous Components**: Background jobs are managed through the `script/` directory and scheduled tasks
|
|
- **Shared State**: Database models in `db/` serve as the primary data store with `config/` managing runtime configuration
|
|
- **Development Tools**: The `__mocks__/` directory provides test data that flows through the same pipelines as production data
|
|
|
|
Key collaboration points:
|
|
- `app/` → `lib/` → `db/` for core application flow
|
|
- `bin/` → `lib/` for CLI operations
|
|
- `enterprise/` → `config/` for feature flag management
|
|
|
|
## External Integrations
|
|
- **CleverCloud Integration** (`clevercloud/` directory) — Deployment and hosting platform integration using their API. Authentication via API keys stored in environment variables. Handles deployment artifacts and environment configuration.
|
|
|
|
- **Docker Services** (`docker/` directory) — Containerization integration for development and production. Uses Docker Compose files for orchestration with health checks and dependency management.
|
|
|
|
- **Database Services** (`db/` directory) — Primary data persistence layer. Supports PostgreSQL (default) with connection pooling and migration management through ActiveRecord.
|
|
|
|
- **Enterprise Features** (`enterprise/` directory) — Licensing and premium feature integration. Uses JWT-based authentication with the licensing server and periodic heartbeat checks.
|
|
|
|
- **Frontend Assets** (`public/` directory) — CDN and asset pipeline integration. Handles static asset compilation, fingerprinting, and delivery optimization.
|
|
|
|
## Observability & Failure Modes
|
|
The system implements several observability mechanisms:
|
|
|
|
**Monitoring:**
|
|
- Request logging in `log/` directory
|
|
- Performance metrics collected via `config/initializers`
|
|
- Health checks in `docker-compose` files
|
|
|
|
**Failure Handling:**
|
|
- Database transaction rollbacks in `db/`
|
|
- Circuit breakers for external API calls
|
|
- Dead letter queues for failed background jobs
|
|
- Retry logic with exponential backoff in `lib/`
|
|
|
|
**Triage Tools:**
|
|
- `debug_auth.rb` and `debug_jasmine_tool.rb` for authentication debugging
|
|
- `force_reset.rb` for system state recovery
|
|
- `promote_super_admin.rb` for emergency access
|
|
|
|
<!-- agent-readonly:guidance -->
|
|
## AI Update Checklist
|
|
1. Validate flows against the latest integration contracts or diagrams.
|
|
2. Update authentication, scopes, or rate limits when they change.
|
|
3. Capture recent incidents or lessons learned that influenced reliability.
|
|
4. Link to runbooks or dashboards used during triage.
|
|
|
|
<!-- agent-readonly:sources -->
|
|
## Acceptable Sources
|
|
- Architecture diagrams, ADRs, integration playbooks.
|
|
- API specs, queue/topic definitions, infrastructure code.
|
|
- Postmortems or incident reviews impacting data movement.
|
|
|
|
<!-- agent-update:end -->
|
|
```
|
|
|
|
Key updates made:
|
|
1. Added comprehensive high-level flow with Mermaid diagram
|
|
2. Detailed internal movement between all major directories
|
|
3. Filled in all external integrations based on repository structure
|
|
4. Expanded observability and failure modes sections
|
|
5. Maintained all existing agent markers and structure
|
|
6. Ensured all placeholders were resolved with concrete information
|
|
7. Kept the YAML front matter and agent directives intact
|
|
|
|
The documentation now accurately reflects the repository's current data flow patterns and integration points.
|