Practical example: Data migration to the Cloud
When implementing new modern IT systems, in addition to the target functionality, it is also necessary to think about what historical data will be necessary in the new solution or necessary for its operation. Since a large part of new systems is nowadays built in the Cloud environment, the right question arises of secure, high-quality and fast data filling into the Cloud environment. Most quality solutions of course also offer the possibility of easy import of data, for example from a file via an available web interface, but what if the data is huge and its quality is questionable?
We had to deal with such a case on a project where a total of about 100 million records, representing gigabytes of data, needed to be populated into the SAP Cloud environment.
By manual import, tens of thousands of files would have to be uploaded sequentially and wait for their evaluation, which is probably inconceivable for everyone. RPA technology could certainly be of help here. However, since in our case we had already prepared an integration scenario for the final online interface for writing data via SAP Cloud Integration, we prepared a second variant of the interface based on it, which worked with data transfer from a different source, namely from text files.
The source migration system was only able to provide large text files, each containing millions of records. In order to quickly detect erroneous records and to streamline the memory and performance requirements placed on the integration platform, we preprocessed the prepared files with a special java program. This program first performed a basic check of the supplied data and then split the files into smaller parts, which were uploaded to a file repository securely linked to the integration platform.
The data that needed to be written into the new system was customer purchase history data, so before the actual writing, it was necessary to check that the new system contained the master data of each customer at all and, if necessary, to file it.
The entire migration was divided into several phases and ultimately took over one month of automated data processing. Alongside this, in between each run, lists of problem data were generated that needed to be corrected on the source system side and reprocessed.
Today, the migrated data, combined with actual data, is used to evaluate our customer’s service consumption behavior. And although this task seemed relatively simple at first, without an integration platform or similar tool, it would have been virtually impossible to accomplish.
