Migration :-(

Ya, Migration was never so tedius, boaring and bugging job for me. It all depends on work you are doing. On the eve of Dipavali, I was all set to run for mumbai and never thought that simple migration will eat my time like anything.


It was a GT migration, as Nilesh was on leave it came on my head. Totally there were 7 workflows needs to be migrated. First time I saw that all workflows under particular folder are getting migrated. Though changes are in only 3 mappings not sure why do they migrate whole code :-(


Each workflow out of 7 has again 12 sessions each. Every sessions was of particular map, so 12 mapping for each session. Out of 12 mappings, 2 mappings were so big, that if you try to open those at Mapping Developer, it was getting hang. Those maps were having 15-16 pipelines each and number of tranformations. yes, that migration was really big.


Around 5 pm, started the migration, and had a plan pack the bags by 6 completing the migration and applying the label.
First Took export of all 7 workflows each and checked for verbose mode. Ya, that thing cs ppl do not like :-(. After getting clear on that, started importing one by one. As changes were not made by me, was very conscious while verifying each and every tranformations, especially targets and source. [THere is a separate nightmare of versioning, will describe it some other time]. For first successful import it took me 20 mins :). After that completed the second workflow which took another 20 mins. Clock's pointer were almost reaching 6, and this time Bikram asked me to call Nilesh again. As was already frustrated, without any hesitation I dialed Nilesh again. and asked him how much time the whole migration usually takes?
He first asked me, how you are doing the migration, one - by - one workflow? and obviouly the answer was Yes.
He started laughing, and said, first take the export by all 7 workflows simultaneously and import only one xml as mostly all trnaformations - source, targets, lookups, filters, expressiosn etc - are common.


* first lesson learned - if you are migrating such large number of workflows, took export of all at same time. As below


At Repository Manager
1. Select all workflows which needs to be migrated and go to repository menu and inside that click on export.

2. Save the wolething as a single xml, give anyname you want. e.g. 7_workFlow_Migration_26Oct.xml


3. Import that xml again, into project or qa folder. 

As you click on import, another window will get popup. Depending upon your requirement, select objects which you want to migrate. In my case, I had selected all objects. and click on next.


Another widow will get appear which will ask you to select the folder - source to destination. Tool is such claver, that it will pick source folder information from xml which you have exported from - hats of to the informaitca coders. For first time, destination folder will be same as source folder name - tool is still not able to read what's in your mind, wait for some more years.


After choosing correct source and destination folders, go ahead by click next. A new Window will ask you whether you want to checkin your new objects in new folder? Now here, Its a belief - its a belief as I never tried it - that if we checkin hear, comments which we give will not appear for checked in objects. So, as per advice by our seniors, please skip this step and do not click on check in.

To resolve the conflicts between objects, new window will appear which will ask you to create any rule if you want. I never came across any such need, so just skip this step and click next.

When you go to next window, here its an important task for decision. You need to resolve the conflicts here either you have to reuse, replace or rename the objects in new folder where you are copying into. In my case, as new folder was totally empty, its showing conflicts only for session properties. 

This is the last step which you will be follwing. and after it you just have to click on import button and all mappings, sessions, wokflow and tranformations inside it will get copied.

The story do not end here actually, after importing all objects, last task was to check-in all the objects with proper comment, so that we can identify what all objects we have imported.

If you observe, here ASA_QA folder is empty, even though we have imported the 3 workflows. Why? because those are not yet checked in. Now, once we check in those all objects, hopfully tool will work fine and will show the objects at QA folder.

give proper comment for objects you are checking in.

Once you checked in those, as per your expectations, tool do not work :). You need to re-connect to the repository and again need to open the QA folder. And now, you can view the things which you are expecting.

But, still my story do not end here, if you have observed, while importing, my all objects were in checked out status, but after importing and checkin in at QA, not a single object was impacted or invalid. Yes that thing you need to verify after you done with the import.

At the time of my migraion, means once I had imported all 7 workflows, one mapping was showing me as impacted. Initially I thought, because of my migration, map got impacted, so I went to map designer, checked it out, did the validation and again checked. I varified the comment just to verify if its been migrated from my user folder, and found the comment was missing. I checked for dependancing now and I could see number of dependant tranformations. "How will you check the dependancy --> just right click on map [or any object for that matter] and click on ' view dependancies..'.

Now I lost my temper, and cursying myself like anything, what went wrong I was not sure. I checked all objects [totally 130+] one by one, noted those which do not have comment which I gave while importing. Searched those missing comments objects at old user folder and found, those objects were not present. Eureka !! Got a clque, might be those sessions and mappings are not connected with those 7 workflows wich I had migrated. So, this time while checking the dependancing, I just click on workflow

Fortune favors the brave, and there were no results :). I had did my job correctly. Only thing I missed is, before importing the objects, I forgot to check if any object is checked-out, impacted or invalid in project or QA folder. Be sure you will verify that...

Now only thing, which was remaining was applying the label.

LABEL -- though not important for developers, but very crucial for ppl who are doing actual migration. Because from Stg --> QA --> Prod migration happens without humam intervation, automatically by tools like pvcs, or pada depending on company.

Go to versioning menu --> inside that Labels ..

After clicking on Label, new window will appear which will ask you to give lavel name which you want to create.

Just give appropriate label name and click on OK. imp: though at image label name is starting with number, it wont allow first character as number and it has to be alphabet. Now, you want to apply label to all workflows which you have migrated.  Thought in a folder, there are only 7 workflows, you are not supposed to add label on whole folder level, reason is, there can be number of objects which are not dependant on 7 workflows, and those will get migrated un-necessarily.

How will you apply label :- Select folder name which contains your workflows, go to Versioning -->Apply Label

Once you click on apply label, new window will appear as below

Click on Next, and window will appear where you want to select the objects for which you want to apply the label.

Here, do not select whole folder, but all required workflows.

Click on Add, and again Next. New windlow will appear where you will actually select the label which you are going to apply.

After selecting the required label, click on OK, give the meaningful comment, and do not forgot to check "Label All Chindren" and "Label All Parents". Click on finish and you are done with your job.

Just to verify, for how many objects label has been applied, you can create a query at informatica level.

Query :-- Go to tool at menubar and inside that click on queries..A new window will appear which will ask you to create new query as below

Give query name you want, select parameter - here I have selected as label, give the respective criteria.

 And shoot on blue arrow. You will get to see all objects which which this label got applied.

 Just verify how many objects you were actually migrated, and for how many objects the label got assigned. Note - label will get assigned only for latest checked in version of an object.

Before I end up - below are the things to always check at the time of migration

1. First verify at ur project/qa or developer folder any object is impacted, checkedout or invalid. If its so, first resolve that and then start for migration.

2. In case of large migration, do not do it one by one, rather take export of all worklows in single go and import all things in single go.

3. Do not checkin object while you are importing it, as comment will not appear in such cases if you do so.

4. After you imported all workflows/objects, please checkin those objects first.

5. Just re-verify if any of the object is impacted, checked out or invalid at project or qa folder.

6. Again write a query on the basis of comment, and count actual number of objects you were migrated and actualy numbr of objects for which comment has been assigned.

7. Once you confirm all above things, apply label only to those objects which you have actually migrated, be careful at this step.

8. Again write a query on basis of applied label and count number of objects again.

Hope this will help you..good luck

Query at ReportExtract

While working on report extract, came across a situation which I thought can not be handled using simple sql query and needs to write pl-sql block; but Sandesh Birwatkar gave a simple solution.

Solution is quite simple, but I never thought of that..

As an example, below is the scenario

You have an employee table, and user will give department number and from which you need to fetch out the respective employees.

select *
from employees
where department_id in (
select distinct department_id
from departments
where department_id = '&DepID'
)

But, twist is, if department table do not have any records, query should return all employee records.

How will you do it? below is the solution

create table departmentsBKP
as
select * from departments

select *
from employees
where department_id in (
select distinct department_id
from departmentsBKP
where department_id = '&DepID'
)
union
select *
from employees
where 0 = (select count(*) from departmentsBKP)

delete from departmentsBKP

and now again if u fire select statement, you will get all records of employees table.

jQuery Test

Hola

This will be hidden slowly.

This will be hidden quickly.

This will not be hidden.

Use of nls_date_language parameter in Oracle

There are some problems where your theoritical knowledge is not at all important. You need to go through such situation. Read an example below..

You have an application whose users are all over the world. You are processing some files every day i.e. each day a new files comes in and you process that using the application available. Even in that, every day you handle one set of file - say for an example on Monday you handle file A, on Tuesday you handle file B, on Wednesday you handle file C etc.

Now issue here we faced was, at India or Germany or France for a particular day files which was getting picked up by the application was not same as application was picking file at US. We checked the date at both end and it was same. So what was fishing? please read below...
Now question is, how will you identify the day of a week - yaa simple SQL Query

 select to_char(sysdate,'D') from dual;

Looks pretty simple and result is quite obvious, it should be ???

We normally counts our week start day as Monday and Week end day as Sunday. so if today is 7-Jan-2009 which is wednesday, the above query will or should return 3 i.e. thid day of a week.

But this does not ends here...Please go through below queries

I have checked the nls_territory parameter value of my db, which results into AMERICA.

SQL> connect
Enter user-name: sys as sysdba
Enter password:
Connected.
SQL> show parameter nls_territory;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
nls_territory                        string      AMERICA
SQL>
SQL> select to_char(sysdate,'D') from dual;
T
-
4

Above query gave me the result as 4, so I varified the sysdate.

SQL>  select to_char(sysdate, 'dd-mon-yyyy HH:MI:SS AM') from dual;
TO_CHAR(SYSDATE,'DD-MON'
-----------------------
07-jan-2009 01:26:14 AM

SQL> select to_char(sysdate,'D') from dual;
T
-
4
SQL>

SQL> select to_char(sysdate, 'Day') from dual;
TO_CHAR(S
---------
Wednesday
SQL>

So, it showed me that, its a 7-Jan-2009, Wednesday and Week Day is 4

Now, I changed my parameter - nls_territory - value to FRANCE

SQL> ALTER SESSION SET NLS_TERRITORY = France;
Session altered.

I again verify my above values

SQL> select to_char(sysdate,'D') from dual;
T
-
3

SQL> select sysdate from dual;
SYSDATE
--------
07/01/09

SQL> select to_char(sysdate, 'dd-mon-yyyy HH:MI:SS') from dual;
TO_CHAR(SYSDATE,'DD-'
--------------------
07-jan-2009 01:22:06

SQL> select to_char(sysdate, 'Day') from dual;
TO_CHAR(S
---------
Wednesday

What does it means?

For 7-Jan-2009, wednesday, if US is a territory Day of a week is 4 but if FRANCE as a territory Day of a week is 3; mmm quite weired, isnt it?


We can try it out using nls_date_language..


SQL> alter session set nls_date_language=GERMAN;
Session altered.

SQL> select to_char(sysdate, 'Day') from dual;
TO_CHAR(SY
----------
Mittwoch

SQL> alter session set nls_date_language=AMERICA;
ERROR:
ORA-12705: Cannot access NLS data files or invalid environment specified

SQL> alter session set nls_date_language=AMERICAN;
Session altered.

SQL> select to_char(sysdate, 'Day') from dual;
TO_CHAR(S
---------
Wednesday
SQL>


This means that, even if we try using Day instead of D, results are different in both cases i.e at US and at Germany.

Try using below now...

SQL> alter session set nls_date_language = GERMAN;
Session altered.

Solution to this problem is, when you are calculating such values always use below to_char function

SQL> select to_char(sysdate, 'Day', 'nls_date_language = GERMAN') from dual;
TO_CHAR(SY
----------
Mittwoch

SQL> select to_char(sysdate, 'Day', 'nls_date_language = AMERICAN') from dual;
TO_CHAR(S
---------
Wednesday
SQL>

Yesss, this works,

So, the thumb rule is, where ever you are located, if you are working with date and timezones are different, be specific in forming a query and use nls_date_language of your query same globally. as mentioned below.

We do not know, why week day at US starts from Sunday and rest of the world from Monday.

Logic behind Number to Date conversion at Excel....

Number of time we have encountered, that some times our number are getting showed in date format and we simply change the format and read it as number as required.


Today while going through the code, I realized the need to understand the logic behind it.


Our source files comes in Excel format, wherein Date column (say as Purchase Date) comes in various formats e.g. 1-Jan-2011, 01-01-2011, 01-January-2011, 01/01/2011, 1-Jan-11 and 40544. All formats are readable except last one 40544.


Now question for me was, how can I convert this 40544 to date? How will my code understand that this is a number and what way should I convert this? Fortunately logic to decode this was available in other mapping.


How do the do it?
They first take out the length of a string (its a date but comes as a string port), first validates all the mentioned formats mentioned above (except number format) and at last if the nothing is true and if length is less than 6 - not sure why they check for less then 6 date; might be they want to stick only to 21st century - they apply below logic.


If we write 1 as a number of excel cell and convert that to date format it appears as 1/1/1900


i.e. for number 1 it considers as 1-Jan-1900. So if we specify 2 it will be 2-Jan-1900. Logic is we need to add that many number of days to 1-Jan-1900. If number is 40544 means 40543 days after 1-Jan-1900. This addition of 1 is not always true as there might be some leap years comes in and changing the date. If we negate 2, logic works fine


Below query will give the results in oracle --


select to_char(to_date('1-Jan-1900','dd-mon-yyyy') + 40544 - 2,'dd-mon-yyyy')  from dual


Query above will give result as 02-jan-2111, which is mathcing with that of excel sheet.

In informatica the logic is as below

ADD_TO_DATE(TO_DATE('01/01/1900', 'dd/mm/yyyy'),'D',TO_INTEGER(NAME))

Add_to_Date will take 3 inputs,
1. Date in which you want to add
2. What you want to add - D/DD/Day means day. M/MM means month and Y/YY means year. Please see informatica help for further information
3. Last is how many numbers you want to add.

XML Source

Today after discussion with my PM , got a question in mind - how its working exactly. Though "Web Services Consumer" tranformation is not getting created at my end because of license issue, I had number of things to check for XML Tranformation

 To start with I have taken XML Source 

To create a XML source, you need to have a XML file. I have taken it from -- http://www.w3schools.com/xml/xml_tree.asp

<?xml version="1.0" encoding="ISO-8859-1"?>
<note>
  <to>Tove</to>
  <from>Jani</from>
  <heading>Reminder</heading>
  <body>Don't forget me this weekend!</body>
</note>

Saved it at Local drive and imported XML Tranformation. While importing it, we get below options

1.You will come across such screen at the time of importing a XML definition. Keep every thing as default.

Sometimes last option "How would you like to generate names for the XML Columns" will be default as first option, select it as second option and check all the mentioned check boxes.

After clicking on OK, you will get a screen asking "How would you like the XML Views to be generated?" as specified below.

Choose any option of Entitiy Relationship and Hierarchy Relationship; structure will be same in this case. Structure may differ for rest of the option or the XML is of more complex nature.

Once you import the tranformation successfully, it will be like this.

Here, structure got pivoted. Note is the main node, that became the primary key of a xml tranformation, rest to, from, heading and body became port of the tranformation i.e. column of a table. So, main outer tag became the row and inner tags became column ultimately.

We will be connecting this XML Source to Target as CSV file.

Below is the structure of a mapping 

This structure is pretty simple, only caution is at the time of connection definition at Session Level. You need to define the XML and Target file path cautiously - please see below

As in the image, we have given value of 2 parameters

Source File Directory = D:\InformaticaFiles\

Source File Name = myXML.xml

By same way you need to specify directory and name for target file.

Once you run the map, target file will have one row inserted as below.

Below is the out-put of target file. Note, while generating the target, create dummy csv file with 5 columns specified with 30 as a each column size.

=================================================================


Now we added 2 more pipelines - so there are not totally 3 pipelines. First will have Normalized XML, second will have De-Normalized and the last one is created with option "Do not generate XML View". We will see regarding how to generate XML Source using "Do not generate XML View option", but if you notice at all 3 source definition, first 2 has one extra port - XPK_X_Note - which is also a primary key. This extra port is missing at 3rd definition which is created with "Do not generate XML View option". We will discuss on how to create XML Source using Do not generate xml view option later.


When we define same target file for all 3 targets, only 1 record get inserted - might be every pipeline will update the same record. So, we defined 3 separate files for each target, first 2 files have unique value for <node> got inserted, third file is missing that particular column.


First port - offcouse for first 2 sources - will generate a unique number for every run. In this case - second pipeline was not exists for first few runs. So, second XML Source count is slightly lacking behind the first source. What we have to find out is, if we dis-connect from repository, still that value persist?

Hmm, dis-connected from repository and after re-connecting it ran the workflow. Concluded values are persistent, might be getting saved at database against workflow run. Not sure how to re-set these values.

One more experiment I did was, I changed the XML Source file now as below


<?xml version="1.0"?>
<note>
    <to>Tove</to>
    <from>Jani</from>
    <heading>Reminder</heading>
    <body>Don't forget me this weekend!</body>
    <to>staksale</to>
    <from>magogate</from>
    <heading>Reminder</heading>
    <body>Meet tomorrow morning at 10</body>
    <to>Tove</to>
    <from>asjaiwa</from>
    <heading>magogate</heading>
    <body>Meet tomorrow morning at 10</body>
</note>


Now there are totally 3 records inside <node>, but when we run the workflow, it saved only first record of Jani.

This means that, our XML Source is defined to read only below structure


<note>
  <to>Tove</to>
  <from>Jani</from>
  <heading>Reminder</heading>
  <body>Don't forget me this weekend!</body>
</note>

If you want to insert number of records, you need to pass that many xml files separately. That's the reason why people are using XML Generator tranformation. So that you just pass your input values to  it, and it will generate the XML dynamically and forward it ahead.

Now we changed the XML files as below


<?xml version="1.0"?>
<note>
    <to>staksale</to>
    <from>magogate</from>
    <heading>Reminder</heading>
    <body>Meet tomorrow morning at 10</body>
    <to>Tove</to>
    <from>Jani</from>
    <heading>Reminder</heading>
    <body>Don't forget me this weekend!</body>   
    <to>Tove</to>
    <from>asjaiwa</from>
    <heading>magogate</heading>
    <body>Meet tomorrow morning at 10</body>
</note>


This time, mapping picked up first record which is of staksale. and you could see it got inserted into source files.


if you want to insert all 3 records, please define 3 separate files for every records, and give it as an input.

Changing data types of columns...

Today I came across one good example

You have one table say ABC, and in that you have 4 cols which are having data type as varchar2(200).

Requirement is , you need to convert the data types of these columns to Number. This table has a data.

How will you change the data types of these columns ?

Note:- You can not create backup table in production...

----------------

CREATE TABLE ABC(COL1 VARCHAR2(200),COL2 VARCHAR2(200),COL3 VARCHAR2(200),COL4

SET DEFINE OFF;
Insert into HR.ABC
   (COL1, COL2, COL3, COL4)
 Values
   ('1', '1', '1', '1');
Insert into HR.ABC
   (COL1, COL2, COL3, COL4)
 Values
   ('2', '2', '2', '2');
Insert into HR.ABC
   (COL1, COL2, COL3, COL4)
 Values
   ('3', '3', '3', '3');
COMMIT;

---------------

TABLE ABCADD

UPDATE (COL11 NUMBER, COL22 NUMBER, COL33 NUMBER, COL44 NUMBER)
---------------

ABCSET COL44 = TO_NUMBER(COL4)
ALTER

UPDATE ABC SET COL33 = TO_NUMBER(COL3)

UPDATE ABC SET COL22 = TO_NUMBER(COL2)


UPDATE ABC SET COL11 = TO_NUMBER(COL1)
----------------

ALTER  TABLE ABC DROP(COL1, COL2, COL3, COL4)
----------------

ALTER TABLE ABC RENAME column COL44 TO COL4

ALTER TABLE ABC RENAME column COL33 TO COL3

ALTER TABLE ABC RENAME column COL22 TO COL2

ALTER TABLE ABC RENAME column COL11 TO COL1


---------------

1. Here, you need to create dummy columns with number datatype at abc, because you can not create column with same name again.
2. You converted data using to_number from varchar to number and updated the newly added columns
3. You dropped the old columns
4. You rename newly added columns