Release v2.2.2: sync safety hardening
This commit is contained in:
parent
bef5c20c83
commit
e772311a86
@ -38,6 +38,8 @@ GITHUB_TOKEN=your_gitea_token_here
|
|||||||
# =====================================================
|
# =====================================================
|
||||||
# API CONFIGURATION - Production
|
# API CONFIGURATION - Production
|
||||||
# =====================================================
|
# =====================================================
|
||||||
|
# Stack name used by deployment scripts to name containers
|
||||||
|
STACK_NAME=prod
|
||||||
API_HOST=0.0.0.0
|
API_HOST=0.0.0.0
|
||||||
API_PORT=8000
|
API_PORT=8000
|
||||||
API_RELOAD=false
|
API_RELOAD=false
|
||||||
|
|||||||
@ -114,6 +114,9 @@ SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_urlsafe(32))")
|
|||||||
# 5. CORS Origins (production domain)
|
# 5. CORS Origins (production domain)
|
||||||
CORS_ORIGINS=https://hub.bmcnetworks.dk
|
CORS_ORIGINS=https://hub.bmcnetworks.dk
|
||||||
|
|
||||||
|
# 5b. Stack name (used by deployment scripts for container names)
|
||||||
|
STACK_NAME=prod
|
||||||
|
|
||||||
# 6. e-conomic Credentials (hvis relevant)
|
# 6. e-conomic Credentials (hvis relevant)
|
||||||
ECONOMIC_APP_SECRET_TOKEN=xxxxx
|
ECONOMIC_APP_SECRET_TOKEN=xxxxx
|
||||||
ECONOMIC_AGREEMENT_GRANT_TOKEN=xxxxx
|
ECONOMIC_AGREEMENT_GRANT_TOKEN=xxxxx
|
||||||
|
|||||||
@ -50,6 +50,7 @@ DATABASE_URL=postgresql://bmc_hub_prod:din_stærke_password_her@postgres:5432/bm
|
|||||||
SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||||
|
|
||||||
# API
|
# API
|
||||||
|
STACK_NAME=prod
|
||||||
API_PORT=8000
|
API_PORT=8000
|
||||||
CORS_ORIGINS=http://172.16.31.183:8001
|
CORS_ORIGINS=http://172.16.31.183:8001
|
||||||
|
|
||||||
|
|||||||
38
RELEASE_NOTES_v2.2.2.md
Normal file
38
RELEASE_NOTES_v2.2.2.md
Normal file
@ -0,0 +1,38 @@
|
|||||||
|
# BMC Hub v2.2.2 - Sync Safety Release
|
||||||
|
|
||||||
|
**Release Date:** 22. februar 2026
|
||||||
|
|
||||||
|
## 🛡️ Critical Fixes
|
||||||
|
|
||||||
|
### e-conomic Customer Sync Mapping
|
||||||
|
- **Fixed ambiguous matching**: e-conomic sync now matches customers only by `economic_customer_number`
|
||||||
|
- **Removed unsafe fallback in this flow**: CVR/name fallback is no longer used in `/api/v1/system/sync/economic`
|
||||||
|
- **Added conflict-safe behavior**: if multiple local rows share the same `economic_customer_number`, the record is skipped and logged as conflict (no overwrite)
|
||||||
|
- **Improved traceability**: sync logs now include the actual local customer id that was updated/created
|
||||||
|
|
||||||
|
### Settings Sync UX
|
||||||
|
- **Aligned frontend with backend response fields** for vTiger/e-conomic sync summaries
|
||||||
|
- **Improved 2FA error feedback** in Settings sync UI when API returns `403: 2FA required`
|
||||||
|
- **Fixed sync stats request limit** to avoid API validation issues
|
||||||
|
- **Temporarily disabled CVR→e-conomic action** in Settings UI to prevent misleading behavior
|
||||||
|
- **Clarified runtime config source**: sync uses environment variables (`.env`) at runtime
|
||||||
|
|
||||||
|
## 🗄️ Database Safety
|
||||||
|
|
||||||
|
### New Migration
|
||||||
|
- Added migration: `migrations/138_customers_economic_unique_constraint.sql`
|
||||||
|
- Normalizes empty/whitespace `economic_customer_number` values
|
||||||
|
- Adds a partial unique index on non-null `economic_customer_number`
|
||||||
|
- Migration aborts with clear error if duplicates already exist (manual dedupe required before rerun)
|
||||||
|
|
||||||
|
## ⚠️ Deployment Notes
|
||||||
|
|
||||||
|
- Run migration `138_customers_economic_unique_constraint.sql` before enabling broad sync operations in production
|
||||||
|
- If migration fails due to duplicates, deduplicate `customers.economic_customer_number` first, then rerun migration
|
||||||
|
- Existing 2FA API protection remains enabled
|
||||||
|
|
||||||
|
## ✅ Expected Outcome
|
||||||
|
|
||||||
|
- Sync payload and DB target row are now consistent in the e-conomic flow
|
||||||
|
- Incorrect overwrites caused by weak matching strategy are prevented
|
||||||
|
- Future duplicate `economic_customer_number` values are blocked at database level
|
||||||
@ -636,6 +636,10 @@
|
|||||||
<div class="mb-4">
|
<div class="mb-4">
|
||||||
<h5 class="fw-bold mb-1">Data Synkronisering</h5>
|
<h5 class="fw-bold mb-1">Data Synkronisering</h5>
|
||||||
<p class="text-muted mb-0">Synkroniser firmaer og kontakter fra vTiger og e-conomic</p>
|
<p class="text-muted mb-0">Synkroniser firmaer og kontakter fra vTiger og e-conomic</p>
|
||||||
|
<div class="alert alert-info mt-3 mb-0 py-2 px-3 small">
|
||||||
|
<i class="bi bi-info-circle me-2"></i>
|
||||||
|
Sync bruger integration credentials fra <strong>miljøvariabler (.env)</strong> ved runtime.
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Sync Status Cards -->
|
<!-- Sync Status Cards -->
|
||||||
@ -733,16 +737,16 @@
|
|||||||
<i class="bi bi-currency-dollar text-success" style="font-size: 2rem;"></i>
|
<i class="bi bi-currency-dollar text-success" style="font-size: 2rem;"></i>
|
||||||
</div>
|
</div>
|
||||||
<div class="flex-grow-1 ms-3">
|
<div class="flex-grow-1 ms-3">
|
||||||
<h6 class="card-title fw-bold">Sync fra e-conomic</h5>
|
<h6 class="card-title fw-bold">Sync fra e-conomic</h6>
|
||||||
<p class="card-text small text-muted">Hent kundenumre fra e-conomic. Matcher på CVR nummer eller firma navn.</p>
|
<p class="card-text small text-muted">Hent kunder fra e-conomic. Matcher kun på entydigt e-conomic kundenummer.</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="d-grid gap-2">
|
<div class="d-grid gap-2">
|
||||||
<button class="btn btn-success" onclick="syncFromEconomic()" id="btnSyncEconomic">
|
<button class="btn btn-success" onclick="syncFromEconomic()" id="btnSyncEconomic">
|
||||||
<i class="bi bi-download me-2"></i>Sync Firmaer fra e-conomic
|
<i class="bi bi-download me-2"></i>Sync Firmaer fra e-conomic
|
||||||
</button>
|
</button>
|
||||||
<button class="btn btn-outline-success btn-sm" onclick="syncCvrToEconomic()" id="btnSyncCvrEconomic">
|
<button class="btn btn-outline-secondary btn-sm" id="btnSyncCvrEconomic" disabled>
|
||||||
<i class="bi bi-search me-2"></i>Find Manglende CVR i e-conomic
|
<i class="bi bi-pause-circle me-2"></i>CVR→e-conomic midlertidigt deaktiveret
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
<div class="mt-3 small">
|
<div class="mt-3 small">
|
||||||
@ -750,6 +754,10 @@
|
|||||||
<i class="bi bi-info-circle me-2"></i>
|
<i class="bi bi-info-circle me-2"></i>
|
||||||
<span>Sidst synkroniseret: <span id="lastSyncEconomic">Aldrig</span></span>
|
<span>Sidst synkroniseret: <span id="lastSyncEconomic">Aldrig</span></span>
|
||||||
</div>
|
</div>
|
||||||
|
<div class="d-flex align-items-center text-muted mt-1">
|
||||||
|
<i class="bi bi-exclamation-circle me-2"></i>
|
||||||
|
<span>CVR-søgning er slået fra midlertidigt for stabil drift.</span>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@ -3256,7 +3264,7 @@ let syncLog = [];
|
|||||||
|
|
||||||
async function loadSyncStats() {
|
async function loadSyncStats() {
|
||||||
try {
|
try {
|
||||||
const response = await fetch('/api/v1/customers?limit=10000');
|
const response = await fetch('/api/v1/customers?limit=1000');
|
||||||
if (!response.ok) throw new Error('Failed to load customers');
|
if (!response.ok) throw new Error('Failed to load customers');
|
||||||
const data = await response.json();
|
const data = await response.json();
|
||||||
const customers = data.customers || [];
|
const customers = data.customers || [];
|
||||||
@ -3321,6 +3329,25 @@ function addSyncLogEntry(title, message, status = 'info') {
|
|||||||
loadSyncLog();
|
loadSyncLog();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function parseApiError(response, fallbackMessage) {
|
||||||
|
let detailMessage = fallbackMessage;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const errorPayload = await response.json();
|
||||||
|
if (errorPayload && errorPayload.detail) {
|
||||||
|
detailMessage = errorPayload.detail;
|
||||||
|
}
|
||||||
|
} catch (parseError) {
|
||||||
|
// Keep fallback message
|
||||||
|
}
|
||||||
|
|
||||||
|
if (response.status === 403 && detailMessage === '2FA required') {
|
||||||
|
return '2FA kræves for sync API. Aktivér 2FA på din bruger og log ind igen.';
|
||||||
|
}
|
||||||
|
|
||||||
|
return detailMessage;
|
||||||
|
}
|
||||||
|
|
||||||
async function syncFromVtiger() {
|
async function syncFromVtiger() {
|
||||||
const btn = document.getElementById('btnSyncVtiger');
|
const btn = document.getElementById('btnSyncVtiger');
|
||||||
btn.disabled = true;
|
btn.disabled = true;
|
||||||
@ -3334,16 +3361,16 @@ async function syncFromVtiger() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
const error = await response.json();
|
const errorMessage = await parseApiError(response, 'Sync fejlede');
|
||||||
throw new Error(error.detail || 'Sync fejlede');
|
throw new Error(errorMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
const result = await response.json();
|
const result = await response.json();
|
||||||
const details = [
|
const details = [
|
||||||
`Behandlet: ${result.total_processed || 0}`,
|
`Behandlet: ${result.total_processed || 0}`,
|
||||||
`Oprettet: ${result.created || 0}`,
|
`Linket: ${result.linked || 0}`,
|
||||||
`Opdateret: ${result.updated || 0}`,
|
`Opdateret: ${result.updated || 0}`,
|
||||||
`Sprunget over: ${result.skipped || 0}`
|
`Ikke fundet/sprunget over: ${result.not_found || 0}`
|
||||||
].join(' | ');
|
].join(' | ');
|
||||||
addSyncLogEntry(
|
addSyncLogEntry(
|
||||||
'vTiger Sync Fuldført',
|
'vTiger Sync Fuldført',
|
||||||
@ -3377,8 +3404,8 @@ async function syncVtigerContacts() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
const error = await response.json();
|
const errorMessage = await parseApiError(response, 'Sync fejlede');
|
||||||
throw new Error(error.detail || 'Sync fejlede');
|
throw new Error(errorMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
const result = await response.json();
|
const result = await response.json();
|
||||||
@ -3418,16 +3445,17 @@ async function syncFromEconomic() {
|
|||||||
});
|
});
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
const error = await response.json();
|
const errorMessage = await parseApiError(response, 'Sync fejlede');
|
||||||
throw new Error(error.detail || 'Sync fejlede');
|
throw new Error(errorMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
const result = await response.json();
|
const result = await response.json();
|
||||||
const details = [
|
const details = [
|
||||||
`Behandlet: ${result.total_processed || 0}`,
|
`Behandlet: ${result.total_processed || 0}`,
|
||||||
`Nye matchet: ${result.matched || 0}`,
|
`Oprettet: ${result.created || 0}`,
|
||||||
`Verificeret: ${result.verified || 0}`,
|
`Opdateret: ${result.updated || 0}`,
|
||||||
`Ikke matchet: ${result.not_matched || 0}`
|
`Konflikter: ${result.conflicts || 0}`,
|
||||||
|
`Sprunget over: ${result.skipped || 0}`
|
||||||
].join(' | ');
|
].join(' | ');
|
||||||
addSyncLogEntry(
|
addSyncLogEntry(
|
||||||
'e-conomic Sync Fuldført',
|
'e-conomic Sync Fuldført',
|
||||||
@ -3449,6 +3477,9 @@ async function syncFromEconomic() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function syncCvrToEconomic() {
|
async function syncCvrToEconomic() {
|
||||||
|
showNotification('CVR→e-conomic sync er midlertidigt deaktiveret.', 'info');
|
||||||
|
return;
|
||||||
|
|
||||||
const btn = document.getElementById('btnSyncCvrEconomic');
|
const btn = document.getElementById('btnSyncCvrEconomic');
|
||||||
btn.disabled = true;
|
btn.disabled = true;
|
||||||
btn.innerHTML = '<span class="spinner-border spinner-border-sm me-2"></span>Søger...';
|
btn.innerHTML = '<span class="spinner-border spinner-border-sm me-2"></span>Søger...';
|
||||||
|
|||||||
@ -24,7 +24,7 @@ HUB owns (manual or first-sync only):
|
|||||||
SYNC RULES:
|
SYNC RULES:
|
||||||
===========
|
===========
|
||||||
1. NEVER overwrite source ID if already set (vtiger_id, economic_customer_number)
|
1. NEVER overwrite source ID if already set (vtiger_id, economic_customer_number)
|
||||||
2. Match order: CVR → Source ID → Name (normalized)
|
2. Matching is source-specific (e-conomic: strict economic_customer_number only)
|
||||||
3. Re-sync is idempotent - can run multiple times safely
|
3. Re-sync is idempotent - can run multiple times safely
|
||||||
4. Contact relationships are REPLACED on sync (not added)
|
4. Contact relationships are REPLACED on sync (not added)
|
||||||
5. Each sync only updates fields it owns
|
5. Each sync only updates fields it owns
|
||||||
@ -475,9 +475,11 @@ async def sync_from_economic() -> Dict[str, Any]:
|
|||||||
created_count = 0
|
created_count = 0
|
||||||
updated_count = 0
|
updated_count = 0
|
||||||
skipped_count = 0
|
skipped_count = 0
|
||||||
|
conflict_count = 0
|
||||||
|
|
||||||
for eco_customer in economic_customers:
|
for eco_customer in economic_customers:
|
||||||
customer_number = eco_customer.get('customerNumber')
|
customer_number_raw = eco_customer.get('customerNumber')
|
||||||
|
customer_number = str(customer_number_raw).strip() if customer_number_raw is not None else None
|
||||||
cvr = eco_customer.get('corporateIdentificationNumber')
|
cvr = eco_customer.get('corporateIdentificationNumber')
|
||||||
name = eco_customer.get('name', '').strip()
|
name = eco_customer.get('name', '').strip()
|
||||||
address = eco_customer.get('address', '')
|
address = eco_customer.get('address', '')
|
||||||
@ -508,20 +510,27 @@ async def sync_from_economic() -> Dict[str, Any]:
|
|||||||
# Extract email domain
|
# Extract email domain
|
||||||
email_domain = email.split('@')[-1] if '@' in email else None
|
email_domain = email.split('@')[-1] if '@' in email else None
|
||||||
|
|
||||||
# Check if customer exists by economic_customer_number OR CVR
|
# Strict matching: ONLY match by economic_customer_number
|
||||||
existing = execute_query(
|
existing = execute_query(
|
||||||
"SELECT id FROM customers WHERE economic_customer_number = %s",
|
"SELECT id, name FROM customers WHERE economic_customer_number = %s ORDER BY id",
|
||||||
(customer_number,)
|
(customer_number,)
|
||||||
)
|
)
|
||||||
|
|
||||||
# If not found by customer number, try CVR (to avoid duplicates)
|
# Conflict handling: duplicate local rows for same e-conomic number
|
||||||
if not existing and cvr:
|
if len(existing) > 1:
|
||||||
existing = execute_query(
|
conflict_count += 1
|
||||||
"SELECT id FROM customers WHERE cvr_number = %s",
|
skipped_count += 1
|
||||||
(cvr,)
|
duplicate_ids = ", ".join(str(row['id']) for row in existing)
|
||||||
|
logger.error(
|
||||||
|
"❌ Konflikt: e-conomic #%s matcher %s lokale kunder (ids: %s) - springer over",
|
||||||
|
customer_number,
|
||||||
|
len(existing),
|
||||||
|
duplicate_ids
|
||||||
)
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
if existing:
|
if existing:
|
||||||
|
target_customer_id = existing[0]['id']
|
||||||
# Update existing customer - ONLY update fields e-conomic owns
|
# Update existing customer - ONLY update fields e-conomic owns
|
||||||
# E-conomic does NOT overwrite: name, cvr_number (set once only)
|
# E-conomic does NOT overwrite: name, cvr_number (set once only)
|
||||||
update_query = """
|
update_query = """
|
||||||
@ -537,10 +546,16 @@ async def sync_from_economic() -> Dict[str, Any]:
|
|||||||
WHERE id = %s
|
WHERE id = %s
|
||||||
"""
|
"""
|
||||||
execute_query(update_query, (
|
execute_query(update_query, (
|
||||||
customer_number, email_domain, address, city, zip_code, country, website, existing[0]['id']
|
customer_number, email_domain, address, city, zip_code, country, website, target_customer_id
|
||||||
))
|
))
|
||||||
updated_count += 1
|
updated_count += 1
|
||||||
logger.info(f"✏️ Opdateret: {name} (e-conomic #{customer_number}, CVR: {cvr or 'ingen'})")
|
logger.info(
|
||||||
|
"✏️ Opdateret lokal kunde id=%s: %s (e-conomic #%s, CVR: %s)",
|
||||||
|
target_customer_id,
|
||||||
|
name,
|
||||||
|
customer_number,
|
||||||
|
cvr or 'ingen'
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
# Create new customer from e-conomic
|
# Create new customer from e-conomic
|
||||||
insert_query = """
|
insert_query = """
|
||||||
@ -555,17 +570,32 @@ async def sync_from_economic() -> Dict[str, Any]:
|
|||||||
))
|
))
|
||||||
|
|
||||||
if result:
|
if result:
|
||||||
|
new_customer_id = result[0]['id']
|
||||||
created_count += 1
|
created_count += 1
|
||||||
logger.info(f"✨ Oprettet: {name} (e-conomic #{customer_number}, CVR: {cvr or 'ingen'})")
|
logger.info(
|
||||||
|
"✨ Oprettet lokal kunde id=%s: %s (e-conomic #%s, CVR: %s)",
|
||||||
|
new_customer_id,
|
||||||
|
name,
|
||||||
|
customer_number,
|
||||||
|
cvr or 'ingen'
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
skipped_count += 1
|
skipped_count += 1
|
||||||
|
|
||||||
logger.info(f"✅ e-conomic sync fuldført: {created_count} oprettet, {updated_count} opdateret, {skipped_count} sprunget over af {len(economic_customers)} totalt")
|
logger.info(
|
||||||
|
"✅ e-conomic sync fuldført: %s oprettet, %s opdateret, %s konflikter, %s sprunget over af %s totalt",
|
||||||
|
created_count,
|
||||||
|
updated_count,
|
||||||
|
conflict_count,
|
||||||
|
skipped_count,
|
||||||
|
len(economic_customers)
|
||||||
|
)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"status": "success",
|
"status": "success",
|
||||||
"created": created_count,
|
"created": created_count,
|
||||||
"updated": updated_count,
|
"updated": updated_count,
|
||||||
|
"conflicts": conflict_count,
|
||||||
"skipped": skipped_count,
|
"skipped": skipped_count,
|
||||||
"total_processed": len(economic_customers)
|
"total_processed": len(economic_customers)
|
||||||
}
|
}
|
||||||
|
|||||||
40
migrations/138_customers_economic_unique_constraint.sql
Normal file
40
migrations/138_customers_economic_unique_constraint.sql
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
-- Migration: Enforce unique economic customer number on customers
|
||||||
|
-- Prevents ambiguous mapping during e-conomic sync
|
||||||
|
|
||||||
|
-- Normalize values before uniqueness check
|
||||||
|
UPDATE customers
|
||||||
|
SET economic_customer_number = NULL
|
||||||
|
WHERE economic_customer_number IS NOT NULL
|
||||||
|
AND btrim(economic_customer_number) = '';
|
||||||
|
|
||||||
|
UPDATE customers
|
||||||
|
SET economic_customer_number = btrim(economic_customer_number)
|
||||||
|
WHERE economic_customer_number IS NOT NULL;
|
||||||
|
|
||||||
|
-- Abort migration if duplicates exist (must be manually resolved first)
|
||||||
|
DO $$
|
||||||
|
DECLARE
|
||||||
|
duplicate_value TEXT;
|
||||||
|
duplicate_count INTEGER;
|
||||||
|
BEGIN
|
||||||
|
SELECT economic_customer_number, COUNT(*)::INTEGER
|
||||||
|
INTO duplicate_value, duplicate_count
|
||||||
|
FROM customers
|
||||||
|
WHERE economic_customer_number IS NOT NULL
|
||||||
|
GROUP BY economic_customer_number
|
||||||
|
HAVING COUNT(*) > 1
|
||||||
|
ORDER BY COUNT(*) DESC, economic_customer_number
|
||||||
|
LIMIT 1;
|
||||||
|
|
||||||
|
IF duplicate_value IS NOT NULL THEN
|
||||||
|
RAISE EXCEPTION 'Cannot create unique index on customers.economic_customer_number. Duplicate value: % (count=%). Resolve duplicates and rerun migration.', duplicate_value, duplicate_count;
|
||||||
|
END IF;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
-- Enforce uniqueness for non-null values
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS customers_economic_customer_number_unique_idx
|
||||||
|
ON customers (economic_customer_number)
|
||||||
|
WHERE economic_customer_number IS NOT NULL;
|
||||||
|
|
||||||
|
COMMENT ON INDEX customers_economic_customer_number_unique_idx IS
|
||||||
|
'Ensures e-conomic customer numbers are unique to prevent sync mismatch.';
|
||||||
Loading…
Reference in New Issue
Block a user