際際滷

際際滷Share a Scribd company logo

What is happening now

D
DreamMalar

The document describes the configuration details of a scheduler that runs jobs to export social media data from various sources to a universal profile database. It outlines the deployment environment, database connections, configurable jobs, and MySQL tables used to track scheduled jobs and export status. The system job checks for new jobs every 20 minutes and exports occur based on cron expressions for each job. Failure details are captured in a MongoDB database.

1 of 4
Download to read offline
Scheduler Architecture 1. Deployment details :- For a QA system the scheduler instance is running on DEV1 server box. Deployment Path - /mnt1/scheduler/ Start script location - /mnt1/scheduler/bin/ Log File Configuration - /mnt1/scheduler/logconfig Spring Configuration - /mnt1/scheduler/spring Log File location - /mnt1/logs/scheduler2.0-global-logs.log Quartz Database - All the Quartz related table are created under the following details. DB Host = DEV1 DB Name = SCHEDULER_GRL:3306 DB user = sprmys DB Password = R1k1rP5 Quartz Tables = QRTZ* Job Details Database - The configured jobs and the last run details are captured in the following tables DB Host = QA2 DB Name = SPR_GRL SCHEDULABLE_JOB_TBL = configured jobs CLIENT_UNIVERSAL_PROFILE_EXPORT_TBL = details last successful runs. Failure Details - At times the Scheduler, Job or the Trigger may fail, these details are captured in the mongo DB. Mongo Host = ec2-107-22-29-36.compute-1.amazonaws.com DB Name = SPR_GLOBAL Collection Names :- failedJobDetail - Captures the failed job details failedSchedulerDetail - Captures the failed scheduler details. triggerMissFireDetail - Captures the missed trigger details.
2. System Job Details :- Sprinklr has been pre-configured with system job which runs every 5 minutes to check the SCHEDULABLE_JOB_TBL changes in the SPR_GRL database. If this time expression to run every five minutes needs to be changed the following entry needs to be made in the spring-quartz.xml file (maked in bold). Adding the following entry would ensure that the system job is scheduled to run every 20 mins as opposed to the 5 minutes it has been configured to run. Restart the scheduler instance after the change has been made to reflect the same. <bean id="sprScheduler" class="com.spr.scheduler.QuartzSchedulerImpl"> <property name="scheduler"> <ref bean="schedulerFactoryBean"/> </property> <property name="jobBasePackageName"> <value>com.spr.jobs</value> </property> <property name="overrideSystemJobMap"> <map> <entry key="com.spr.jobs.system.JobSchedulerJob"> <!-- scheduled the job to run once a century at every 20 mins --> <value>* 0/20 * * * ?</value> </entry> </map> </property> <ignored rest of the properties/>
3. Configurable Jobs :- The following list of Jobs can be configured in the scheduler. 1. com.spr.jobs.export.twitter.ExportTwitterClientProfileToUniversalProfileJob Exports the data from PROFILE_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to then universalProfile collection. 2. com.spr.jobs.export.twitter.ExportTwitterConversationToUniversalProfileJob Exports the data from ACCOUNT_TWEET_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to then conversation collection. 3. com.spr.jobs.export.twitter.ExportTwitterFollowerAndFollowingDetailsJob Export the data from ACCOUNT_FFUF_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the universalProfile collection. 4. com.spr.jobs.export.facebook.ExportFacebookClientProfileToUniversalProfileJob Export the data from FB_PROFILE_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the universalProfile collection. 5. com.spr.jobs.export.facebook.ExportFacebookCommentsToUniversalProfileJob Exports the data from FB_ACCOUNT_COMMENTS_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the conversation collection. 6. com.spr.jobs.export.facebook.ExportFacebookConversationToUniversalProfileJob Exports the data from FB_INBOX_STREAM_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the conversation collection.
4. MySql Tables :- SCHEDULABLE_JOB_TBL - This table captures the job which need to be scheduled in the scheduler. The JOB_CLASS, PARTNER_ID, CLIENT_ID and the CLIENT_ID form the uniqueness. SCHEDULABLE_JOB_TBL_ID - The primary key JOB_CLASS_NAME - The name of job class, as defined above. This is the business logic to executes. PARTNER_ID - The Partner Id CLIENT_ID - The Client Id ACCOUNT_ID - The Account Id JOB_DATA - Any additional data as a JSON string to be made available to job during its execution. CRON_EXPRESSION - The Cron Expression which defines the run intervals. IS_SCHEDULED - Boolean to define is the job is scheduled on the scheduler. This column is used by the System Job and is internal to the scheduler. IS_ENABLED - if set to true, would enable the job in the scheduler to run on the scheduler cron time, if set to false, would disable the enabled job from the scheduler, and this job will not be qualified for any further runs. CREATED_TM - The time at which this record was created MODIFIED_TM - The time at which this record was updated. DEL_FLG - If set to true, would delete the job from the scheduler along with its associated triggers. To qualify this job to run again set the flag to false. SCHEDULE_KEY - The unique key which identifies the job. This column is used by the System Job and is internal to the scheduler. CLIENT_UNIVERSAL_PROFILE_EXPORT_TBL - This table captures the last run details for the job configured. This table is internal to the jobs and should be used only for look- ups. PARTNER_ID - The Partner id CLIENT_ID - The Client Id ACCOUNT_ID - The account Id LAST_EXPORT_DATE - The date till which the job did its last export. SN_TYPE - The Social Network Type EXPORT_TYPE - The Export type, can be one of Profile, Conversation or Comment

More Related Content

What is happening now

  • 1. Scheduler Architecture 1. Deployment details :- For a QA system the scheduler instance is running on DEV1 server box. Deployment Path - /mnt1/scheduler/ Start script location - /mnt1/scheduler/bin/ Log File Configuration - /mnt1/scheduler/logconfig Spring Configuration - /mnt1/scheduler/spring Log File location - /mnt1/logs/scheduler2.0-global-logs.log Quartz Database - All the Quartz related table are created under the following details. DB Host = DEV1 DB Name = SCHEDULER_GRL:3306 DB user = sprmys DB Password = R1k1rP5 Quartz Tables = QRTZ* Job Details Database - The configured jobs and the last run details are captured in the following tables DB Host = QA2 DB Name = SPR_GRL SCHEDULABLE_JOB_TBL = configured jobs CLIENT_UNIVERSAL_PROFILE_EXPORT_TBL = details last successful runs. Failure Details - At times the Scheduler, Job or the Trigger may fail, these details are captured in the mongo DB. Mongo Host = ec2-107-22-29-36.compute-1.amazonaws.com DB Name = SPR_GLOBAL Collection Names :- failedJobDetail - Captures the failed job details failedSchedulerDetail - Captures the failed scheduler details. triggerMissFireDetail - Captures the missed trigger details.
  • 2. 2. System Job Details :- Sprinklr has been pre-configured with system job which runs every 5 minutes to check the SCHEDULABLE_JOB_TBL changes in the SPR_GRL database. If this time expression to run every five minutes needs to be changed the following entry needs to be made in the spring-quartz.xml file (maked in bold). Adding the following entry would ensure that the system job is scheduled to run every 20 mins as opposed to the 5 minutes it has been configured to run. Restart the scheduler instance after the change has been made to reflect the same. <bean id="sprScheduler" class="com.spr.scheduler.QuartzSchedulerImpl"> <property name="scheduler"> <ref bean="schedulerFactoryBean"/> </property> <property name="jobBasePackageName"> <value>com.spr.jobs</value> </property> <property name="overrideSystemJobMap"> <map> <entry key="com.spr.jobs.system.JobSchedulerJob"> <!-- scheduled the job to run once a century at every 20 mins --> <value>* 0/20 * * * ?</value> </entry> </map> </property> <ignored rest of the properties/>
  • 3. 3. Configurable Jobs :- The following list of Jobs can be configured in the scheduler. 1. com.spr.jobs.export.twitter.ExportTwitterClientProfileToUniversalProfileJob Exports the data from PROFILE_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to then universalProfile collection. 2. com.spr.jobs.export.twitter.ExportTwitterConversationToUniversalProfileJob Exports the data from ACCOUNT_TWEET_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to then conversation collection. 3. com.spr.jobs.export.twitter.ExportTwitterFollowerAndFollowingDetailsJob Export the data from ACCOUNT_FFUF_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the universalProfile collection. 4. com.spr.jobs.export.facebook.ExportFacebookClientProfileToUniversalProfileJob Export the data from FB_PROFILE_TBL based on the MODIFIED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the universalProfile collection. 5. com.spr.jobs.export.facebook.ExportFacebookCommentsToUniversalProfileJob Exports the data from FB_ACCOUNT_COMMENTS_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the conversation collection. 6. com.spr.jobs.export.facebook.ExportFacebookConversationToUniversalProfileJob Exports the data from FB_INBOX_STREAM_TBL based on the CREATED_TM, CLIENT_ID, ACCOUNT_ID and the PARTNER_ID to the conversation collection.
  • 4. 4. MySql Tables :- SCHEDULABLE_JOB_TBL - This table captures the job which need to be scheduled in the scheduler. The JOB_CLASS, PARTNER_ID, CLIENT_ID and the CLIENT_ID form the uniqueness. SCHEDULABLE_JOB_TBL_ID - The primary key JOB_CLASS_NAME - The name of job class, as defined above. This is the business logic to executes. PARTNER_ID - The Partner Id CLIENT_ID - The Client Id ACCOUNT_ID - The Account Id JOB_DATA - Any additional data as a JSON string to be made available to job during its execution. CRON_EXPRESSION - The Cron Expression which defines the run intervals. IS_SCHEDULED - Boolean to define is the job is scheduled on the scheduler. This column is used by the System Job and is internal to the scheduler. IS_ENABLED - if set to true, would enable the job in the scheduler to run on the scheduler cron time, if set to false, would disable the enabled job from the scheduler, and this job will not be qualified for any further runs. CREATED_TM - The time at which this record was created MODIFIED_TM - The time at which this record was updated. DEL_FLG - If set to true, would delete the job from the scheduler along with its associated triggers. To qualify this job to run again set the flag to false. SCHEDULE_KEY - The unique key which identifies the job. This column is used by the System Job and is internal to the scheduler. CLIENT_UNIVERSAL_PROFILE_EXPORT_TBL - This table captures the last run details for the job configured. This table is internal to the jobs and should be used only for look- ups. PARTNER_ID - The Partner id CLIENT_ID - The Client Id ACCOUNT_ID - The account Id LAST_EXPORT_DATE - The date till which the job did its last export. SN_TYPE - The Social Network Type EXPORT_TYPE - The Export type, can be one of Profile, Conversation or Comment