bmc_hub/docs/DATA_CONSISTENCY_SYSTEM.md

# Data Consistency Checking System - Implementation Complete

## 📅 Implementation Date
8. januar 2026

## 🎯 Overview
Implemented a comprehensive data consistency checking system that automatically compares customer data across BMC Hub, vTiger Cloud, and e-conomic when loading a customer detail page. The system detects discrepancies and allows manual selection of the correct value to sync across all systems.

## ✅ Completed Features

### 1. Configuration Variables (`app/core/config.py`)
Added three new boolean flags to the Settings class:
- `VTIGER_SYNC_ENABLED: bool = True` - Enable/disable vTiger sync operations
- `ECONOMIC_SYNC_ENABLED: bool = True` - Enable/disable e-conomic sync operations
- `AUTO_CHECK_CONSISTENCY: bool = True` - Enable/disable automatic consistency checking

### 2. CustomerConsistencyService (`app/services/customer_consistency.py`)
Created a new service with the following functionality:

#### Field Mapping
Maps 11 customer fields across all three systems:
- name → accountname (vTiger) / name (e-conomic)
- cvr_number → cf_856 / corporateIdentificationNumber
- address → bill_street / address
- city → bill_city / city
- postal_code → bill_code / zip
- country → bill_country / country
- phone → phone / telephoneAndFaxNumber
- mobile_phone → mobile / mobilePhone
- email → email1 / email
- website → website / website
- invoice_email → email2 / email

#### Key Methods
- **`normalize_value()`**: Normalizes values for comparison (strip, lowercase, None handling)
- **`fetch_all_data()`**: Fetches customer data from all three systems in parallel using `asyncio.gather()`
- **`compare_data()`**: Compares normalized values and identifies discrepancies
- **`sync_field()`**: Updates a field across all enabled systems with the selected correct value

### 3. Backend API Endpoints (`app/customers/backend/router.py`)
Added two new endpoints:

#### GET `/api/v1/customers/{customer_id}/data-consistency`
- Checks if consistency checking is enabled
- Fetches data from all systems in parallel
- Compares all fields and counts discrepancies
- Returns:
  ```json
  {
    "enabled": true,
    "customer_id": 123,
    "discrepancy_count": 3,
    "discrepancies": {
      "address": {
        "hub": "Hovedgaden 1",
        "vtiger": "Hovedgade 1",
        "economic": "Hovedgaden 1",
        "discrepancy": true
      }
    },
    "systems_available": {
      "hub": true,
      "vtiger": true,
      "economic": false
    }
  }
  ```

#### POST `/api/v1/customers/{customer_id}/sync-field`
- Query parameters:
  - `field_name`: Hub field name (e.g., "address")
  - `source_system`: "hub", "vtiger", or "economic"
  - `source_value`: The correct value to sync
- Updates the field in all systems
- Respects safety flags (ECONOMIC_READ_ONLY, ECONOMIC_DRY_RUN)
- Returns sync status for each system

### 4. Frontend Alert Box (`app/customers/frontend/customer_detail.html`)
Added a Bootstrap warning alert that:
- Displays after customer header when discrepancies are found
- Shows discrepancy count dynamically
- Has a "Sammenlign" (Compare) button to open the modal
- Is dismissible
- Hidden by default with `.d-none` class

### 5. Comparison Modal (`app/customers/frontend/customer_detail.html`)
Created a modal-xl Bootstrap modal with:
- Table showing all discrepant fields
- 5 columns: Felt (Field), BMC Hub, vTiger, e-conomic, Vælg Korrekt (Select Correct)
- Radio buttons for each system's value
- Danish field labels (Navn, CVR Nummer, Adresse, etc.)
- Visual indicators for unavailable systems
- "Synkroniser Valgte" (Sync Selected) button

### 6. JavaScript Functions (`app/customers/frontend/customer_detail.html`)
Implemented three main functions:

#### `checkDataConsistency()`
- Called automatically when customer loads
- Fetches consistency data from API
- Shows/hides alert based on discrepancy count
- Stores data in `consistencyData` global variable

#### `showConsistencyModal()`
- Populates modal table with only discrepant fields
- Creates radio buttons dynamically for each system
- Uses Danish field labels
- Handles unavailable systems gracefully

#### `syncSelectedFields()`
- Collects all selected radio button values
- Validates at least one selection
- Shows confirmation dialog
- Calls sync API for each field sequentially
- Shows success/failure count
- Reloads customer data and rechecks consistency

### 7. VTiger Service Updates (`app/services/vtiger_service.py`)
Added two new methods:
- **`get_account_by_id(account_id)`**: Fetches single account by vTiger ID
- **`update_account(account_id, update_data)`**: Updates account fields via REST API

### 8. E-conomic Service Updates (`app/services/economic_service.py`)
Added two new methods:
- **`get_customer(customer_number)`**: Fetches single customer by e-conomic number
- **`update_customer(customer_number, update_data)`**: Updates customer fields (respects safety flags)

## 🔧 Technical Implementation Details

### Parallel API Calls
Uses `asyncio.gather()` with `return_exceptions=True` to fetch from vTiger and e-conomic simultaneously:
```python
tasks = {'vtiger': vtiger_task, 'economic': economic_task}
task_results = await asyncio.gather(*tasks.values(), return_exceptions=True)
```

### Value Normalization
All values are normalized before comparison:
- Convert to string
- Strip whitespace
- Lowercase for case-insensitive comparison
- Empty strings → None

### Safety Flags
Respects existing e-conomic safety flags:
- `ECONOMIC_READ_ONLY=True` prevents all write operations
- `ECONOMIC_DRY_RUN=True` logs operations without executing

### Error Handling
- Individual system failures don't block the entire operation
- Exceptions are logged with appropriate emoji prefixes (✅ ❌ ⚠️)
- Frontend shows user-friendly messages

## 🎨 User Experience

### Workflow
1. User opens customer detail page (e.g., `/customers/23`)
2. `loadCustomer()` automatically calls `checkDataConsistency()`
3. If discrepancies found, yellow alert appears at top
4. User clicks "Sammenlign" button
5. Modal opens showing table with radio buttons
6. User selects correct value for each field
7. User clicks "Synkroniser Valgte"
8. Confirmation dialog appears
9. Selected fields sync to all systems
10. Page reloads with updated data

### Visual Design
- Uses existing Nordic Top design system
- Bootstrap 5 components (alerts, modals, tables)
- Consistent with BMC Hub's minimalist aesthetic
- Danish language throughout

## 📝 Configuration

### Environment Variables (.env)
```bash
# Data Consistency Settings
VTIGER_SYNC_ENABLED=True
ECONOMIC_SYNC_ENABLED=True
AUTO_CHECK_CONSISTENCY=True

# Safety Flags (respect existing)
ECONOMIC_READ_ONLY=True
ECONOMIC_DRY_RUN=True
```

### Disabling Features
- Set `AUTO_CHECK_CONSISTENCY=False` to disable automatic checks
- Set `VTIGER_SYNC_ENABLED=False` to prevent vTiger updates
- Set `ECONOMIC_SYNC_ENABLED=False` to prevent e-conomic updates

## 🚀 Deployment

### Status
✅ **DEPLOYED AND RUNNING**

The system has been:
1. Code implemented in all necessary files
2. Docker API container restarted successfully
3. Service running without errors (confirmed via logs)
4. Ready for testing at http://localhost:8001/customers/{id}

### Testing Checklist
- [ ] Open customer detail page
- [ ] Verify alert appears if discrepancies exist
- [ ] Click "Sammenlign" and verify modal opens
- [ ] Select values and click "Synkroniser Valgte"
- [ ] Confirm data syncs across systems
- [ ] Verify safety flags prevent unwanted writes

## 📚 Files Modified

1. `/app/core/config.py` - Added 3 config variables
2. `/app/services/customer_consistency.py` - NEW FILE (280 lines)
3. `/app/customers/backend/router.py` - Added 2 endpoints (~100 lines)
4. `/app/customers/frontend/customer_detail.html` - Added alert, modal, and JS functions (~250 lines)
5. `/app/services/vtiger_service.py` - Added 2 methods (~90 lines)
6. `/app/services/economic_service.py` - Added 2 methods (~75 lines)

**Total new code:** ~795 lines

## 🎓 Key Learnings

1. **Parallel async operations** are essential for performance when querying multiple external APIs
2. **Data normalization** is critical for accurate comparison (whitespace, case sensitivity, null handling)
3. **Progressive enhancement** - system degrades gracefully if external APIs are unavailable
4. **Safety-first approach** - dry-run and read-only flags prevent accidental data corruption
5. **User-driven sync** - manual selection ensures humans make final decisions on data conflicts

## 🔮 Future Enhancements

Potential improvements for future iterations:
- Auto-suggest most common value (modal default selection)
- Batch sync all fields with single button
- Conflict history log
- Scheduled consistency checks (background job)
- Email notifications for critical discrepancies
- Automatic sync rules (e.g., "always trust e-conomic for financial data")
- Conflict resolution confidence scores

## ✅ Implementation Complete

All planned features have been successfully implemented and deployed. The data consistency checking system is now active and ready for use.

**Next Steps:** Test the system with real customer data to ensure all integrations work correctly.