Batch processing Transnational Data Suggestions


hello,

i have requirement data comes in flat file format.  this data need go through various step of transformations , lookup . have broke down process in steps.  1. stage 2. collect needed data 3. validate data 4. store them in incremental order future use. 5. generate more transaction related incoming rows. 6 send target system.  

through out process data going across many tables , servers , have following requirement.

1. need create process in case of failure  if data has not gone through steps  and in target system need  process start left off , finish processing data target.  ( requirement ask not use check point in ssis)

2. in case failure require process same file again data should wipe cleaned in step 1 5 , new file should process  

i have looked in batch processing in steps batch id there need better suggestions. please let me know if have suggestion or route should avoid.

i advise against using transactions long transaction not advisable.

however using audit tables capture load success updated when tasks completed.

if success do:

  1. archive source files
  2. update load status success

if failure:

  1. leave file in source
  2. update load status fail

when job run next time, check status above, if success start new process, else run same process again.

note: need have way capture file names processed , failures rerun. maintain in log table along loadid. on next run check files load id not success , run jobs files in foreah loop if need.

my blog    |      ask me     |      ssis basics     



SQL Server  >  SQL Server Integration Services



Comments

Post a Comment

Popular posts from this blog

BIT Version

  hi,     knows how find out ther bit version in sql  , windows os (32 or 64)   regards, s.balavenkatesh   microsoft sql server 2005 - 9.00.1399.06 (intel x86) oct 14 2005 00:33:37 copyright (c) 1988-2005 microsoft corporation standard edition on windows nt 5.2 (build 3790: service pack 2)     intel x86)   -- says sql server 32 bit   madhu SQL Server  >  SQL Server Tools
Read more

SQL Server PSProvider SQL Server Authentication

greetings: i'm trying use sql psprovider powershell.  can make work management studio by right-clicking next database object.  however, unnable external powershell session. note: i have tried using suggested method msdn ( initializesqlprovider.ps1 , function sqldrive ) i following error: new-psdrive : sql server powershell provider error: not connect 'emtyb06\default+bca_empresarial'. [object ref erence not set instance of object.] @ line:1 char:12 + new-psdrive  <<<< "sqlauth" -psprovider sqlserver -root "sqlserver:\sql\emtyb06\default+bca_empresarial" -credential $cred -scope 1 new-psdrive : sql server powershell provider error: path sqlserver:\sql\emtyb06\default+bca_empresarial not exist.  please specify valid path. @ line:1 char:12 + new-psdrive  <<<< "sqlauth" -psprovider sqlserver -root "sqlserver:\sql\emtyb06\default+bca_empresarial" -cred...
Read more

Getting error when using Excel Destination

Image
i getting error when using   transferring data oledb source  to excel destination i using dynamic file connection means each time new file generated @ desination folder getting error given above path stored in variable @[user::path] :c/admin/ for creating excel file using expression excelfile path: @[user::path] +(dt_str,4,1252) datepart("yyyy",getdate()) +right("0"+(dt_str,4,1252) datepart("m",getdate()),2) +right("0" + (dt_str,4,1252) datepart("d",getdate()),2)+ right("0" + (dt_str,4,1252) datepart("hh",getdate()),2)+right("0" + (dt_str,4,1252) datepart("mm",getdate()),2)+right("0" + (dt_str,4,1252) datepart("ss",getdate()),2)+".xls" now when clicking on exceldestination component mapping getting error ================================= error @ insertingdatafromexcel de...
Read more