Renaming columns using geopandas - issues with data table in Arc





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







1















What I'm trying to achieve:




  • Read in an existing ESRI polygon shapefile

  • Drop an attribute

  • Dissolve by another attribute

  • Rename columns

  • Reorder columns

  • Write out as new shapefile


What works: Drop column and dissolve all work - I've run these steps and written to file. This opens fine in ArcMap, I can identify things, open the attribute table etc. No issues.



Problem: When I rename (not got to the reorder step without error yet) the columns, everything runs fine and I write to file. When then opening the shapefile in ArcMap (10.3.1) I can query layers using the Identify cursor which shows my renamed attributes as they should be BUT when trying to open the attribute table, I get the following message in a window:



Loading table data



Could not load data from the data source. If you can correct the problem, press the refresh button to reload data. Possible problems can include bad network connection, invalid file, etc.



A column was specified that doesn't exists.



A column was specified that doesn't exists.



So, it looks like something is missing or my changes are not being mapped across the shapefile components.



Given a mock version of the data:



# Make a mock geopandas dataframe
import geopandas as gpd
from shapely.geometry import Point, Polygon
from fiona.crs import from_epsg

data = gpd.GeoDataFrame()

# Coordinates of some locations in British National Grid (epsg: 27700)
coordinates1 = [(19.950899, 60.169158), (20.953492, 60.169158), (20.950958, 61.169990), (19.953510, 61.170104)]
coordinates2 = [(21.950899, 60.169158), (22.953492, 60.169158), (22.950958, 61.169990), (21.953510, 61.170104)]
coordinates3 = [(19.99, 60.3), (20.23492, 60.3), (20.23492, 60.8), (19.99, 60.8)]

# Create a Shapely polygon from the coordinate-tuple list
poly1 = Polygon(coordinates1)
poly2 = Polygon(coordinates2)
poly3 = Polygon(coordinates3) # (sits inside polygon 1)

# Add polygon to dataframe
data.loc[0, 'geometry'] = poly1
data.loc[1, 'geometry'] = poly2
data.loc[2, 'geometry'] = poly3
data.loc[0, 'value'] = 1
data.loc[1, 'value'] = 2
data.loc[2, 'value'] = 3
data.loc[0, 'CLASS'] = 'A'
data.loc[1, 'CLASS'] = 'B'
data.loc[2, 'CLASS'] = 'A'
data.loc[0, 'Hazard_Pot'] = 'High'
data.loc[1, 'Hazard_Pot'] = 'Low'
data.loc[2, 'Hazard_Pot'] = 'High'
data["contaminat"]="something"

# Define CRS
data.crs = from_epsg(27700)


## Plot it
#import matplotlib.pyplot as plt
#ax1=data.plot()
#data.loc[[2],'geometry'].plot(ax=ax1, color='red') # plot on top of others
#plt.show()


We'll now dissolve the polygons based on Hazard_Pot and then rename and reorder things. The problem is with the renaming... What am I missing?



# Drop a column (if writing out file here, does as expected)
data=data.drop(['CLASS'], axis=1)

# Do the dissolve (if writing out file here, does as expected)
data=data.dissolve(by='Hazard_Pot', as_index=False)

# Rename <<< won't work in ArcGIS
data=data.rename(index=str, columns={"Hazard_Pot":"Potential",
"contaminat":"Contam."}) # <<< EDIT: notice the punctuation (noticed following the accepted answer)

# Reorder
data=data[['value',
'Contam.',
'Potential',
'geometry']]

# write to file (creating a new CRS based on the old as per https://gis.stackexchange.com/questions/204201/geopandas-to-file-saves-geodataframe-without-coordinate-system?noredirect=1&lq=1)
ofile='./rename_shape.shp'
data.to_file(ofile, driver='ESRI Shapefile')









share|improve this question

























  • data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

    – Paul H
    Mar 27 at 15:45











  • I can't share the data but can share the code and you can share fake data that reproduces the issue

    – Paul H
    Mar 27 at 15:46











  • @PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

    – BERA
    Mar 27 at 16:10













  • @BERA Oops - forgot rename could take callables.

    – Paul H
    Mar 27 at 16:38


















1















What I'm trying to achieve:




  • Read in an existing ESRI polygon shapefile

  • Drop an attribute

  • Dissolve by another attribute

  • Rename columns

  • Reorder columns

  • Write out as new shapefile


What works: Drop column and dissolve all work - I've run these steps and written to file. This opens fine in ArcMap, I can identify things, open the attribute table etc. No issues.



Problem: When I rename (not got to the reorder step without error yet) the columns, everything runs fine and I write to file. When then opening the shapefile in ArcMap (10.3.1) I can query layers using the Identify cursor which shows my renamed attributes as they should be BUT when trying to open the attribute table, I get the following message in a window:



Loading table data



Could not load data from the data source. If you can correct the problem, press the refresh button to reload data. Possible problems can include bad network connection, invalid file, etc.



A column was specified that doesn't exists.



A column was specified that doesn't exists.



So, it looks like something is missing or my changes are not being mapped across the shapefile components.



Given a mock version of the data:



# Make a mock geopandas dataframe
import geopandas as gpd
from shapely.geometry import Point, Polygon
from fiona.crs import from_epsg

data = gpd.GeoDataFrame()

# Coordinates of some locations in British National Grid (epsg: 27700)
coordinates1 = [(19.950899, 60.169158), (20.953492, 60.169158), (20.950958, 61.169990), (19.953510, 61.170104)]
coordinates2 = [(21.950899, 60.169158), (22.953492, 60.169158), (22.950958, 61.169990), (21.953510, 61.170104)]
coordinates3 = [(19.99, 60.3), (20.23492, 60.3), (20.23492, 60.8), (19.99, 60.8)]

# Create a Shapely polygon from the coordinate-tuple list
poly1 = Polygon(coordinates1)
poly2 = Polygon(coordinates2)
poly3 = Polygon(coordinates3) # (sits inside polygon 1)

# Add polygon to dataframe
data.loc[0, 'geometry'] = poly1
data.loc[1, 'geometry'] = poly2
data.loc[2, 'geometry'] = poly3
data.loc[0, 'value'] = 1
data.loc[1, 'value'] = 2
data.loc[2, 'value'] = 3
data.loc[0, 'CLASS'] = 'A'
data.loc[1, 'CLASS'] = 'B'
data.loc[2, 'CLASS'] = 'A'
data.loc[0, 'Hazard_Pot'] = 'High'
data.loc[1, 'Hazard_Pot'] = 'Low'
data.loc[2, 'Hazard_Pot'] = 'High'
data["contaminat"]="something"

# Define CRS
data.crs = from_epsg(27700)


## Plot it
#import matplotlib.pyplot as plt
#ax1=data.plot()
#data.loc[[2],'geometry'].plot(ax=ax1, color='red') # plot on top of others
#plt.show()


We'll now dissolve the polygons based on Hazard_Pot and then rename and reorder things. The problem is with the renaming... What am I missing?



# Drop a column (if writing out file here, does as expected)
data=data.drop(['CLASS'], axis=1)

# Do the dissolve (if writing out file here, does as expected)
data=data.dissolve(by='Hazard_Pot', as_index=False)

# Rename <<< won't work in ArcGIS
data=data.rename(index=str, columns={"Hazard_Pot":"Potential",
"contaminat":"Contam."}) # <<< EDIT: notice the punctuation (noticed following the accepted answer)

# Reorder
data=data[['value',
'Contam.',
'Potential',
'geometry']]

# write to file (creating a new CRS based on the old as per https://gis.stackexchange.com/questions/204201/geopandas-to-file-saves-geodataframe-without-coordinate-system?noredirect=1&lq=1)
ofile='./rename_shape.shp'
data.to_file(ofile, driver='ESRI Shapefile')









share|improve this question

























  • data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

    – Paul H
    Mar 27 at 15:45











  • I can't share the data but can share the code and you can share fake data that reproduces the issue

    – Paul H
    Mar 27 at 15:46











  • @PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

    – BERA
    Mar 27 at 16:10













  • @BERA Oops - forgot rename could take callables.

    – Paul H
    Mar 27 at 16:38














1












1








1


0






What I'm trying to achieve:




  • Read in an existing ESRI polygon shapefile

  • Drop an attribute

  • Dissolve by another attribute

  • Rename columns

  • Reorder columns

  • Write out as new shapefile


What works: Drop column and dissolve all work - I've run these steps and written to file. This opens fine in ArcMap, I can identify things, open the attribute table etc. No issues.



Problem: When I rename (not got to the reorder step without error yet) the columns, everything runs fine and I write to file. When then opening the shapefile in ArcMap (10.3.1) I can query layers using the Identify cursor which shows my renamed attributes as they should be BUT when trying to open the attribute table, I get the following message in a window:



Loading table data



Could not load data from the data source. If you can correct the problem, press the refresh button to reload data. Possible problems can include bad network connection, invalid file, etc.



A column was specified that doesn't exists.



A column was specified that doesn't exists.



So, it looks like something is missing or my changes are not being mapped across the shapefile components.



Given a mock version of the data:



# Make a mock geopandas dataframe
import geopandas as gpd
from shapely.geometry import Point, Polygon
from fiona.crs import from_epsg

data = gpd.GeoDataFrame()

# Coordinates of some locations in British National Grid (epsg: 27700)
coordinates1 = [(19.950899, 60.169158), (20.953492, 60.169158), (20.950958, 61.169990), (19.953510, 61.170104)]
coordinates2 = [(21.950899, 60.169158), (22.953492, 60.169158), (22.950958, 61.169990), (21.953510, 61.170104)]
coordinates3 = [(19.99, 60.3), (20.23492, 60.3), (20.23492, 60.8), (19.99, 60.8)]

# Create a Shapely polygon from the coordinate-tuple list
poly1 = Polygon(coordinates1)
poly2 = Polygon(coordinates2)
poly3 = Polygon(coordinates3) # (sits inside polygon 1)

# Add polygon to dataframe
data.loc[0, 'geometry'] = poly1
data.loc[1, 'geometry'] = poly2
data.loc[2, 'geometry'] = poly3
data.loc[0, 'value'] = 1
data.loc[1, 'value'] = 2
data.loc[2, 'value'] = 3
data.loc[0, 'CLASS'] = 'A'
data.loc[1, 'CLASS'] = 'B'
data.loc[2, 'CLASS'] = 'A'
data.loc[0, 'Hazard_Pot'] = 'High'
data.loc[1, 'Hazard_Pot'] = 'Low'
data.loc[2, 'Hazard_Pot'] = 'High'
data["contaminat"]="something"

# Define CRS
data.crs = from_epsg(27700)


## Plot it
#import matplotlib.pyplot as plt
#ax1=data.plot()
#data.loc[[2],'geometry'].plot(ax=ax1, color='red') # plot on top of others
#plt.show()


We'll now dissolve the polygons based on Hazard_Pot and then rename and reorder things. The problem is with the renaming... What am I missing?



# Drop a column (if writing out file here, does as expected)
data=data.drop(['CLASS'], axis=1)

# Do the dissolve (if writing out file here, does as expected)
data=data.dissolve(by='Hazard_Pot', as_index=False)

# Rename <<< won't work in ArcGIS
data=data.rename(index=str, columns={"Hazard_Pot":"Potential",
"contaminat":"Contam."}) # <<< EDIT: notice the punctuation (noticed following the accepted answer)

# Reorder
data=data[['value',
'Contam.',
'Potential',
'geometry']]

# write to file (creating a new CRS based on the old as per https://gis.stackexchange.com/questions/204201/geopandas-to-file-saves-geodataframe-without-coordinate-system?noredirect=1&lq=1)
ofile='./rename_shape.shp'
data.to_file(ofile, driver='ESRI Shapefile')









share|improve this question
















What I'm trying to achieve:




  • Read in an existing ESRI polygon shapefile

  • Drop an attribute

  • Dissolve by another attribute

  • Rename columns

  • Reorder columns

  • Write out as new shapefile


What works: Drop column and dissolve all work - I've run these steps and written to file. This opens fine in ArcMap, I can identify things, open the attribute table etc. No issues.



Problem: When I rename (not got to the reorder step without error yet) the columns, everything runs fine and I write to file. When then opening the shapefile in ArcMap (10.3.1) I can query layers using the Identify cursor which shows my renamed attributes as they should be BUT when trying to open the attribute table, I get the following message in a window:



Loading table data



Could not load data from the data source. If you can correct the problem, press the refresh button to reload data. Possible problems can include bad network connection, invalid file, etc.



A column was specified that doesn't exists.



A column was specified that doesn't exists.



So, it looks like something is missing or my changes are not being mapped across the shapefile components.



Given a mock version of the data:



# Make a mock geopandas dataframe
import geopandas as gpd
from shapely.geometry import Point, Polygon
from fiona.crs import from_epsg

data = gpd.GeoDataFrame()

# Coordinates of some locations in British National Grid (epsg: 27700)
coordinates1 = [(19.950899, 60.169158), (20.953492, 60.169158), (20.950958, 61.169990), (19.953510, 61.170104)]
coordinates2 = [(21.950899, 60.169158), (22.953492, 60.169158), (22.950958, 61.169990), (21.953510, 61.170104)]
coordinates3 = [(19.99, 60.3), (20.23492, 60.3), (20.23492, 60.8), (19.99, 60.8)]

# Create a Shapely polygon from the coordinate-tuple list
poly1 = Polygon(coordinates1)
poly2 = Polygon(coordinates2)
poly3 = Polygon(coordinates3) # (sits inside polygon 1)

# Add polygon to dataframe
data.loc[0, 'geometry'] = poly1
data.loc[1, 'geometry'] = poly2
data.loc[2, 'geometry'] = poly3
data.loc[0, 'value'] = 1
data.loc[1, 'value'] = 2
data.loc[2, 'value'] = 3
data.loc[0, 'CLASS'] = 'A'
data.loc[1, 'CLASS'] = 'B'
data.loc[2, 'CLASS'] = 'A'
data.loc[0, 'Hazard_Pot'] = 'High'
data.loc[1, 'Hazard_Pot'] = 'Low'
data.loc[2, 'Hazard_Pot'] = 'High'
data["contaminat"]="something"

# Define CRS
data.crs = from_epsg(27700)


## Plot it
#import matplotlib.pyplot as plt
#ax1=data.plot()
#data.loc[[2],'geometry'].plot(ax=ax1, color='red') # plot on top of others
#plt.show()


We'll now dissolve the polygons based on Hazard_Pot and then rename and reorder things. The problem is with the renaming... What am I missing?



# Drop a column (if writing out file here, does as expected)
data=data.drop(['CLASS'], axis=1)

# Do the dissolve (if writing out file here, does as expected)
data=data.dissolve(by='Hazard_Pot', as_index=False)

# Rename <<< won't work in ArcGIS
data=data.rename(index=str, columns={"Hazard_Pot":"Potential",
"contaminat":"Contam."}) # <<< EDIT: notice the punctuation (noticed following the accepted answer)

# Reorder
data=data[['value',
'Contam.',
'Potential',
'geometry']]

# write to file (creating a new CRS based on the old as per https://gis.stackexchange.com/questions/204201/geopandas-to-file-saves-geodataframe-without-coordinate-system?noredirect=1&lq=1)
ofile='./rename_shape.shp'
data.to_file(ofile, driver='ESRI Shapefile')






python shapefile geopandas






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 28 at 8:50







ChrisWills

















asked Mar 27 at 15:40









ChrisWillsChrisWills

506




506













  • data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

    – Paul H
    Mar 27 at 15:45











  • I can't share the data but can share the code and you can share fake data that reproduces the issue

    – Paul H
    Mar 27 at 15:46











  • @PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

    – BERA
    Mar 27 at 16:10













  • @BERA Oops - forgot rename could take callables.

    – Paul H
    Mar 27 at 16:38



















  • data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

    – Paul H
    Mar 27 at 15:45











  • I can't share the data but can share the code and you can share fake data that reproduces the issue

    – Paul H
    Mar 27 at 15:46











  • @PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

    – BERA
    Mar 27 at 16:10













  • @BERA Oops - forgot rename could take callables.

    – Paul H
    Mar 27 at 16:38

















data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

– Paul H
Mar 27 at 15:45





data.rename(index=str... renaming the index to a data type is very strange. Take that part out and report back

– Paul H
Mar 27 at 15:45













I can't share the data but can share the code and you can share fake data that reproduces the issue

– Paul H
Mar 27 at 15:46





I can't share the data but can share the code and you can share fake data that reproduces the issue

– Paul H
Mar 27 at 15:46













@PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

– BERA
Mar 27 at 16:10







@PaulH i belive index=str is correct syntax, see the examples in help section of rename. But I also think that part is not needed.

– BERA
Mar 27 at 16:10















@BERA Oops - forgot rename could take callables.

– Paul H
Mar 27 at 16:38





@BERA Oops - forgot rename could take callables.

– Paul H
Mar 27 at 16:38










1 Answer
1






active

oldest

votes


















4














Looking at your code I can see two things that Arcmap doesn't like:




  • The first one is the space in the field name "building d";

  • The second one - and the cause of the error - is the . in "contaminat":"Contam.".


I was able to reproduce the error by creating a test file and using your code. Removing the dot "contaminat":"Contam" solved the problem.






share|improve this answer
























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "79"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f316876%2frenaming-columns-using-geopandas-issues-with-data-table-in-arc%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    4














    Looking at your code I can see two things that Arcmap doesn't like:




    • The first one is the space in the field name "building d";

    • The second one - and the cause of the error - is the . in "contaminat":"Contam.".


    I was able to reproduce the error by creating a test file and using your code. Removing the dot "contaminat":"Contam" solved the problem.






    share|improve this answer




























      4














      Looking at your code I can see two things that Arcmap doesn't like:




      • The first one is the space in the field name "building d";

      • The second one - and the cause of the error - is the . in "contaminat":"Contam.".


      I was able to reproduce the error by creating a test file and using your code. Removing the dot "contaminat":"Contam" solved the problem.






      share|improve this answer


























        4












        4








        4







        Looking at your code I can see two things that Arcmap doesn't like:




        • The first one is the space in the field name "building d";

        • The second one - and the cause of the error - is the . in "contaminat":"Contam.".


        I was able to reproduce the error by creating a test file and using your code. Removing the dot "contaminat":"Contam" solved the problem.






        share|improve this answer













        Looking at your code I can see two things that Arcmap doesn't like:




        • The first one is the space in the field name "building d";

        • The second one - and the cause of the error - is the . in "contaminat":"Contam.".


        I was able to reproduce the error by creating a test file and using your code. Removing the dot "contaminat":"Contam" solved the problem.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Mar 27 at 16:23









        Mouad AlamiMouad Alami

        1637




        1637






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Geographic Information Systems Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f316876%2frenaming-columns-using-geopandas-issues-with-data-table-in-arc%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Biblatex bibliography style without URLs when DOI exists (in Overleaf with Zotero bibliography)

            ComboBox Display Member on multiple fields

            Is it possible to collect Nectar points via Trainline?