Array combination for HNCO (12m + 7m + TP)

[d-l-walker]

This is general issue for tracking and discussing the 12m + 7m + TP combination for the spectral line data.
HNCO data is being prioritised initially.
Below is a checklist of regions that have been completed so far, along with links to download the combined products.

Note: the hyperlinks don't appear to be working, so for now you can just copy and past the link address, which should work. Products will soon be migrated to our Globus repo.

See the comments for this issue below for more info & images.

HNCO

• Sgr_A_st_a
  • Sgr_A_st_a_12m_7m_TP.HNCO.fits
  • Sgr_A_st_a_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_aa
  • Sgr_A_st_aa_12m_7m_TP.HNCO.fits
  • Sgr_A_st_aa_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_ac
  • Sgr_A_st_ac_12m_7m_TP.HNCO.fits
  • Sgr_A_st_ac_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_ag
  • Sgr_A_st_ag_12m_7m_TP.HNCO.fits
  • Sgr_A_st_ag_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_ai
  • Sgr_A_st_ai_12m_7m_TP.HNCO.fits
  • Sgr_A_st_ai_12m_7m.clean.image.HNCO.fits
• Sgr_A_st_ak
  • Sgr_A_st_ak_12m_7m_TP.HNCO.fits
  • Sgr_A_st_ak_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_ao
  • Sgr_A_st_ao_12m_7m_TP.HNCO.fits
  • Sgr_A_st_ao_12m_7m.clean.image.HNCO.fits
• Sgr_A_st_ap
  • Sgr_A_st_ap_12m_7m.HNCO.clean.image.fits
  • Note: TP data not yet observed/delivered. Feathered product will be added here once available.
• Sgr_A_st_c
  • Sgr_A_st_c_12m_7m_TP.HNCO.fits
  • Sgr_A_st_c_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_d
  • Sgr_A_st_d_12m_7m_TP.HNCO.fits
  • Sgr_A_st_d_12m_7m.clean.image.HNCO.fits
• Sgr_A_st_h
  • Sgr_A_st_h_12m_7m_TP.HNCO.fits
  • Sgr_A_st_h_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_i
  • Sgr_A_st_i_12m_7m_TP.HNCO.fits
  • Sgr_A_st_i_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_j
  • Sgr_A_st_j_12m_7m.HNCO.clean.image.fits
  • Note: Velocity-corrected TP data not yet delivered. Feathered product will be added here once available.
• Sgr_A_st_l
  • Sgr_A_st_l_12m_7m_TP.HNCO.fits
  • Sgr_A_st_l_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_m
  • Sgr_A_st_m_12m_7m_TP.HNCO.fits
  • Sgr_A_st_m_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_n
  • Sgr_A_st_n_12m_7m_TP.HNCO.fits
  • Sgr_A_st_n_12m_7m.HNCO.clean.image.fits
• Sgr_A_st_w
  • Sgr_A_st_w_12m_7m_TP.HNCO.fits
  • Sgr_A_st_w_12m_7m.clean.image.HNCO.fits

[d-l-walker]

Peak HNCO intensity maps for Sgr_A_st_ao

Left: 12m + 7m + TP, Right: 12m + 7m
(See #122 for more discussion on this region)

Peak HNCO intensity maps for Sgr_A_st_w. Lots of emission!

Left: 12m + 7m + TP, Right: 12m + 7m

[d-l-walker]

For reference, here are the scripts that I'm currently using to do this:
https://github.com/d-l-walker/ACES_personal/tree/main/joint_deconvolution

This is a work-in-progress and can almost certainly be improved, streamlined, packaged up properly, etc. This is just a first attempt. I'm already aware of a few fixes/exceptions etc. that I need to implement.

Right now I'm just running a master shell script to execute the relevant scripts in-turn. I just pass a region argument to the python scripts (e.g. Sgr_A_st_ao), and the rest is automated.

The process is a bit slow at the moment, as I'm having to download the relevant datasets locally and set up the directory structures. I hope that soon we should have the large ACES NAS drives set up at the UK ARC Node, where we can mirror the data hosted on Globus. Once this is in place I should be able to set things up for that fixed directory structure, and better automate this to run on all regions for which we have 12m + 7m + TP data delivered.

[keflavich]

@d-l-walker could you put your joint_deconvolution directory/project into the main repository? It can sit around as a "draft" pull request for as long as you like, but this is a perfect case to be able to have code review.

[d-l-walker]

@keflavich Done: see #195

I'm happy for it to be merged if that would make the most sense. I'll just update with any changes as I continue to make improvements.

[d-l-walker]

The first fully-automated end-to-end run of the scripts just finished. This was for region Sgr_A_st_ai. Everything went smoothly and took just over 2 days to finish, but the 12m+7m cleaning diverged across a few channels due to bright edge emission.

The HNCO peak intensity of the 12m+7m+TP still looks decent (left), but you can see the divergence in the cube towards the centre of the lower edge (right). I was already running with cyclefactor=1.5 as my default. I've bumped this up to 2.5 to see whether that fixes things.

[ashleythomasbarnes]

@d-l-walker - could you also produce a HNCO cube where you just feather the 12m with 7m and then feather 12m+7m with the tp? And then compare with your joint deconvolution?

[d-l-walker]

I've added 2 more completed regions to this issue. Peak intensity maps shown below. Link to download the FITS cubes can be found at the top of this issue.

Peak HNCO intensity maps for Sgr_A_st_a (#242)

Left: 12m + 7m + TP, Right: 12m + 7m

Peak HNCO intensity maps for Sgr_A_st_aa (#251)

Left: 12m + 7m + TP, Right: 12m + 7m

[d-l-walker]

Some more finished regions. The links to download the FITS cubes have been added to the main post for this issue.

Peak HNCO intensity maps for Sgr_A_st_ac (#232)

Left: 12m + 7m + TP, Right: 12m + 7m

Peak HNCO intensity maps for Sgr_A_st_d (#247)

Left: 12m + 7m + TP, Right: 12m + 7m

Peak HNCO intensity maps for Sgr_A_st_ai (#130, this one was posted above noting divergence, which is now fixed)

Left: 12m + 7m + TP, Right: 12m + 7m

[d-l-walker]

Another region finished: Sgr_A_st_ak (#120)

• Looks like the addition of the TP data is making a much more significant contribution here, compared to the previous fields.
• Download links added to the parent comment.

[d-l-walker]

Another region finished: Sgr_A_st_ap (#43)

• Note: we don't have the TP data for this field yet, so only the 12m + 7m combination has been done for this field. I will add in the TP data as soon as we receive it.
• Download link has been added to the top.

[d-l-walker]

Another region finished: Sgr_A_st_c (#256). Download links have been added to the parent comment for this issue.

This field also includes the Brick. Probably worthwhile cleaning together with field ao (#41) at some point.

[d-l-walker]

Region Sgr_A_st_ag (#136) finished. Download links have been added to the parent comment for this issue.

[d-l-walker]

Region Sgr_A_st_i (#138) finished.
The peak emission looks really nice in this region (which is just below Sgr B2 south). The TP data is also making a clear difference.
Will add download links above shortly.

[d-l-walker]

Region Sgr_A_st_j (#117) finished.
This is also just below Sgr B2 (below North/Main).
Peak intensity maps look great, and again the TP data is adding a lot.

For now, I will only upload the 12m + 7m data, as the TP data we have in-hand has the wrong velocity shift. The updated SB has not yet been delivered. Once it has, I'll re-do the feathering and upload the fully combined cube.

[d-l-walker]

12m + 7m HNCO combination for region k (#152) diverged, despite using cyclefactor=3.0. It also looks like the imsize, which is pulled directly from the PL 12m parameters, is not suitable here. The image is wrapping in the vertical direction. Will try re-running with a higher cyclefactor and a larger image size.

[d-l-walker]

Regions l (#137) and m (#118) have finished. The TP data don't appear to be adding much to region l, but the contribution for region m is more significant. (The latter is Sgr B2 N, hence the bright compact sources.) Download links added to to parent comment as always.

[d-l-walker]

Sgr_A_st_n (#231) finished. Emission looks fairly faint and fluffy here. This region is right at the lower left of the mosaic, so this is probably not too surprising. Adding download links now.

[d-l-walker]

Quick, dirty test following yesterday's discussion re: data combination.

• Left: the jointly imaged 12m + 7m, plus feathered TP data (peak HNCO).
• Centre: the result of feathering 7m + TP data first, then feathering 7m+TP with 12m data (similar to @pyhsiehATalma's approach here: Array combination for CS(2-1) (12m + 7m + TP) #339).
• Right: difference between the two (first - second image).

Both look great, and the feathering-only approach is significantly faster, but there's a clear difference between the two, with the joint imaging + feathered image containing more (peak) intensity.

I just ran a quick, standalone test for this. Next step is to do this exactly as @pyhsiehATalma is doing for the CS data, following her sample code in #341 and #340.

[d-l-walker]

A few more regions have finished.

Region p (#149) looks really nice. I think this may be ~ the 20 km/s cloud? TP data is making a big difference.

For the next few images, the TP combination has not finished yet, so apologies for the crappy screenshots ...

Region u (#241) looks interesting, though relatively diffuse. This region is at the bottom edge of our mosaic, just to the lower right of Sgr B2.

Region v (#235) doesn't seem to show a whole lot of structure. This region is basically ~ 0.15 deg directly above Sgr A*, so maybe this isn't surprising given the latitude.

Region q (#109) looks neat, but there's something funky going on at the edges. My guess is that it'll be something due to the additional channels in the 7m data causing there to be a handful of edge channels with no 12m data. I'll look into this and fix as appropriate. For context, I think this region corresponds to one of the "far side" clouds at negative longitudes (maybe the "wiggles"?).

[d-l-walker]

Regions ah (#238) and g (#134) look super cool. They're adjacent field at the lower left of the mosaic.

Note that these are 12m + 7m + TP, all feathered together (see #350), not jointly imaged like the previous images in this issue.

Next step is to reproject these, plus all other combined HNCO peak intensity maps, into a single mosaic.

[d-l-walker]

First attempt a combined peak HNCO intensity mosaic, and it's looking fantastic! (Open the image in a new tab to appreciate it fully). The whole Sgr B2++ region looks crazy.

There are a bunch of regions missing for various reasons:

• Field af had diverged, so I excluded it for now.
• A few regions are still missing the TP data.
• A few regions simply failed.

Again, note that these are feathered-only maps. I will repeat this for the jointly-images data soon.

[snlongmore]

Wow!! Great work, that is truly spectacular!!!

[d-l-walker]

Updated with a couple more fields.

[d-l-walker]

Updated 12m+7m+TP peak HNCO mosaic.
Only 3 regions missing now (r, y, af), we're getting there!

• The following TP SBs still need to be downloaded: Execution Block ID uid://A001/X15a0/X10a Sgr_A_st_r_03_TP #246 and Execution Block ID uid://A001/X15a0/X134 Sgr_A_st_y_03_TP #27
• The HNCO SPW for Execution Block ID uid://A001/X15a0/X15a Sgr_A_st_af_03_TM1 #255 is still under QA/imaging due to an unusual form of divergence

While we are waiting for these outstanding SBs to complete the mosaic, I'll work on incorporating region weighting as @pyhsiehATalma has done for the CS mosaic in order to suppress the individual mosaic edges.

[ashleythomasbarnes]

Quick comparison between the HNCO 12m+7m+tp (left, blue spec), tp (centre, orange spec), and mopra (right, red spec).
Looks like we are recovering the total power flux pretty well, but there are some differences when compared to the mopra data. When looking through the mopra cube there are clearly baseline issues present. The baseline of the total power cube actually looks pretty nice, but I think there may be some overfitting from the pipeline going on here. Also worth pointing out that the mopra data doesn't exactly cover the lines we observed - e.g. the CS(2-1) isn't included... Anyway I still plan to try feather with the mopra data due to the higher angular resolution, as this could result in a better combination (let's see).

[keflavich]

@pyhsiehATalma has CS 2-1 data from Nobeyama, though - we can compare to those:
https://github.com/CentralMolecularZone/DataSets -> https://zenodo.org/record/7708322#.ZAiKsnZBy5c

[ashleythomasbarnes]

(copying some old stuff from Slack below)

[ashleythomasbarnes]

Ashley Barnes
2 months ago
I downloaded the brick joint clean and it appears to have stronger negative bowls than the feathered image (which actually doesn’t look that bad.. Anyone else finding something similar?

Daniel Walker
2 months ago
I seem to remember the joint imaging of this field was very prone to diverging right in that lower left region, so it’s possible that something’s still dodgy with the jointly imaged cube here (edited)

Ashley Barnes
2 months ago
Oh right, yep, probably not the easiest with the emission so close to the edge of the map

Ashley Barnes
2 months ago
I’ll download another field to check

Daniel Walker
2 months ago
I’ll try re-doing the joint imaging again with some tweaked parameters to hopefully address this. Thanks for catching it!

Ashley Barnes
2 months ago
No rush on this one, maybe would be good to focus on the tests before rerunning
👍
1

Ashley Barnes
2 months ago
I’m downloading feild i now - looks like a good one with lots of extended emission

Ashley Barnes
2 months ago
So it seems like where there is real emission, the joint imaged map does recover more of the larger scales and generally looks a bit nicer. However, it also appears to contain stronger and negative bowls around this emission compared to the feathered image…

[ashleythomasbarnes]

Ashley Barnes
2 months ago
When integrating over a large area comparable to the single dish resolution, I find that be fluxes are more or less the same, which makes me think this isn’t a problem with the TP feathering, but rather the cleaning of the 12m and 7m

Ashley Barnes
2 months ago
The negative bowls are not as prominent in the 12m only image…

Daniel Walker
2 months ago
Yeah we’d noticed this in the past (https://docs.google.com/document/d/1kB1LJBfREhLHWzaw3kKR5E0tpE_T-pC8piQfAI-Injk/edit#bookmark=id.iad7sv37lpi)
It seems that the negatives from the 7m are adding significant negatives into the jointly imaged map, but not when they’re all feathered.

Daniel Walker
2 months ago
To be clear about how the joint imaging was done, all I’m doing is taking the exact tclean parameters that were used for the 12m, and just adding the 7m ms files in the vis parameter. The only other parameter I’ve tweaked is the cyclefactor when necessary.

Ashley Barnes
2 months ago
Ooo I see… hmm okay

Ashley Barnes
2 months ago
Also I’ve been playing with 1) including the MOPRA data to your cleaned 7m+12m images and it isn’t great, there are still a number of strong negative bowls in the image (I haven’t tested if its better than the TP); 2) increasing the weighting factor of the single dish in feather (attached show MOPRA+7m+12m; factors of 1, 2, 3, 4, and 5). You can see that upping the factor does solve the bowls, but some other artefacts are present (and the flux gets too high).

Ashley Barnes
2 months ago
How do the residuals in the 7m+12m cleaned images look
@Daniel Walker
?

Daniel Walker
2 months ago
Sorry
@Ashley Barnes
, I’ve been caught up with other things since last week and haven’t had a lot of time to spend on this.
I’ve just had a quick look at some regions, and in general the residuals look decent. Not perfect, the automasking isn’t quite getting all of the fainter extended emission, but it’s doing a reasonable job considering the complexity.
Screenshots show a single channel comparison, and the mean spectrum comparison.
We’ll have to have a more detailed look at this at some point obviously, this is just at a glance.