2024-09-22 06:33:27,559 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Current SDK version is 0.17.5
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Configure stats pid to 1338204
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Loading settings from /home/yangyaodong/.config/wandb/settings
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Loading settings from /aifs4su/yaodong/projects/hantao/dev_cham/align-anything/scripts/wandb/settings
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Loading settings from environment variables: {'api_key': '***REDACTED***'}
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
2024-09-22 06:33:27,560 WARNING MainThread:1338204 [wandb_setup.py:_flush():76] Could not find program at -m align_anything.trainers.tiv_to_t.dpo
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m align_anything.trainers.tiv_to_t.dpo'}
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_setup.py:_flush():76] Applying login settings: {}
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:_log_setup():529] Logging user logs to ../outputs/dpo_tiv2t_10k_baseline/wandb/run-20240922_063327-rptdqsqq/logs/debug.log
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:_log_setup():530] Logging internal logs to ../outputs/dpo_tiv2t_10k_baseline/wandb/run-20240922_063327-rptdqsqq/logs/debug-internal.log
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:init():569] calling init triggers
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:init():576] wandb.init called with sweep_config: {}
config: {'train_cfgs': {'ds_cfgs': 'ds_z3_config.json', 'epochs': 3, 'seed': 42, 'per_device_train_batch_size': 1.0, 'per_device_eval_batch_size': 1.0, 'gradient_accumulation_steps': 1.0, 'gradient_checkpointing': True, 'learning_rate': 1e-06, 'lr_scheduler_type': 'cosine', 'lr_warmup_ratio': 0.01, 'weight_decay': 0.0, 'adam_betas': [0.9, 0.95], 'bf16': True, 'fp16': False, 'eval_strategy': 'epoch', 'eval_interval': 10, 'regularization': 0.001, 'scale_coeff': 0.1, 'freeze_mm_proj': False, 'freeze_vision_tower': True, 'freeze_language_model': False}, 'data_cfgs': {'train_datasets': '/aifs4su/yaodong/datasets/aaa_dataset/TV2T-preference/extracted', 'train_template': 'NExTQA_preference', 'train_size': None, 'train_split': 'train', 'train_subset': None, 'train_data_files': 'extracted_preference_10k_washed.json', 'train_optional_args': [], 'eval_datasets': None, 'eval_template': None, 'eval_size': None, 'eval_split': None, 'eval_subset': None, 'eval_data_files': None, 'eval_optional_args': []}, 'logger_cfgs': {'log_type': 'wandb', 'log_project': 'align-anything', 'log_run_name': 'dpo', 'output_dir': '../outputs/dpo_tiv2t_10k_baseline', 'cache_dir': None, 'save_interval': 100000}, 'model_cfgs': {'model_name_or_path': '/aifs4su/yaodong/models/Qwen2-VL-7B-Instruct', 'trust_remote_code': True, 'model_max_length': 4096}, 'special_tokens': None}
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:init():619] starting backend
2024-09-22 06:33:27,560 INFO    MainThread:1338204 [wandb_init.py:init():623] setting up manager
2024-09-22 06:33:27,561 INFO    MainThread:1338204 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-09-22 06:33:27,564 INFO    MainThread:1338204 [wandb_init.py:init():631] backend started and connected
2024-09-22 06:33:27,566 INFO    MainThread:1338204 [wandb_init.py:init():720] updated telemetry
2024-09-22 06:33:27,587 INFO    MainThread:1338204 [wandb_init.py:init():753] communicating run to backend with 90.0 second timeout
2024-09-22 06:33:28,008 INFO    MainThread:1338204 [wandb_run.py:_on_init():2435] communicating current version
2024-09-22 06:33:28,205 INFO    MainThread:1338204 [wandb_run.py:_on_init():2444] got version response upgrade_message: "wandb version 0.18.1 is available!  To upgrade, please run:\n $ pip install wandb --upgrade"

2024-09-22 06:33:28,205 INFO    MainThread:1338204 [wandb_init.py:init():804] starting run threads in backend
2024-09-22 06:33:34,310 INFO    MainThread:1338204 [wandb_run.py:_console_start():2413] atexit reg
2024-09-22 06:33:34,310 INFO    MainThread:1338204 [wandb_run.py:_redirect():2255] redirect: wrap_raw
2024-09-22 06:33:34,310 INFO    MainThread:1338204 [wandb_run.py:_redirect():2320] Wrapping output streams.
2024-09-22 06:33:34,310 INFO    MainThread:1338204 [wandb_run.py:_redirect():2345] Redirects installed.
2024-09-22 06:33:34,313 INFO    MainThread:1338204 [wandb_init.py:init():847] run started, returning control to user process
2024-09-22 16:29:07,169 INFO    MainThread:1338204 [wandb_run.py:_finish():2107] finishing run htlou/align-anything/rptdqsqq
2024-09-22 16:29:07,171 INFO    MainThread:1338204 [wandb_run.py:_atexit_cleanup():2374] got exitcode: 0
2024-09-22 16:29:07,172 INFO    MainThread:1338204 [wandb_run.py:_restore():2352] restore
2024-09-22 16:29:07,172 INFO    MainThread:1338204 [wandb_run.py:_restore():2358] restore done
2024-09-22 16:29:15,801 INFO    MainThread:1338204 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2024-09-22 16:29:15,802 INFO    MainThread:1338204 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2024-09-22 16:29:15,809 INFO    MainThread:1338204 [wandb_run.py:_footer_sync_info():3975] logging synced files